Oct 31 00:24:55.241384 ip-10-0-129-8 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 00:24:55.241391 ip-10-0-129-8 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 00:24:55.241398 ip-10-0-129-8 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 00:24:55.241624 ip-10-0-129-8 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:25:05.433368 ip-10-0-129-8 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:25:05.433386 ip-10-0-129-8 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ead03ab65c5f49959c3f8c8102a41b81 -- Oct 31 00:27:23.928987 ip-10-0-129-8 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:27:24.373575 ip-10-0-129-8 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:27:24.373575 ip-10-0-129-8 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:27:24.373575 ip-10-0-129-8 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:27:24.373575 ip-10-0-129-8 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:27:24.379660 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379532 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:24.379660 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379657 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:24.379660 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379661 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:24.379660 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379664 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:24.379660 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379667 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:24.379660 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379669 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379672 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379675 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379677 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379680 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379683 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379686 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379704 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379707 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379710 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379712 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379715 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379717 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379720 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379722 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379725 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379727 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379730 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379732 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379736 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:24.379897 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379742 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379745 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379747 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379750 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379778 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379782 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379786 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379789 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379791 2568 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379794 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379796 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379799 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379802 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379805 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379808 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379811 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379813 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379816 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379818 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379821 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:24.380352 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379823 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379826 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379828 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379835 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379840 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379842 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379845 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379848 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379850 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379853 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379857 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379861 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379864 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379867 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379869 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379872 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379874 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379877 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379879 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:24.380840 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379882 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379884 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379887 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379890 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379893 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379897 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379900 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379902 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379905 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379908 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379910 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379913 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379915 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379918 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379920 2568 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379923 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379925 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379928 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379930 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379934 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:24.381321 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379936 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379939 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379941 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379944 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379946 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379949 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379951 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379954 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379957 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379959 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379961 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379964 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379966 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379969 2568 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379971 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379973 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379976 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379979 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379981 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379984 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:24.381798 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379987 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.379991 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380795 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380802 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380804 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380807 2568 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380810 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380813 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380817 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380821 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380824 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380827 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380830 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380833 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380835 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380838 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380841 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380843 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380846 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:24.382249 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380849 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380851 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380854 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380856 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380859 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380861 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380864 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380866 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380869 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380872 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380875 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380877 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380880 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380883 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380886 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380888 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380890 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380893 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380895 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380898 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:24.382727 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380900 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380903 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380906 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380908 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380911 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380913 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380916 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380919 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380921 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380924 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380926 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380930 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380932 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380935 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380937 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380940 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380942 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380945 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380948 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:24.383201 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380950 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380953 2568 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380955 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380958 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380961 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380964 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380966 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380968 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380971 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380973 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380976 2568 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380978 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380981 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380983 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380986 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380988 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380991 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380993 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380995 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.380998 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:24.383655 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381001 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381003 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381006 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381008 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381011 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381014 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381017 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381019 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381021 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381024 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381029 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381031 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381034 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381036 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381039 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381041 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381044 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381046 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381049 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:24.384169 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381051 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381055 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381059 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381062 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381065 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381067 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381070 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381072 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381074 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381077 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381079 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381667 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381674 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381678 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381682 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381684 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381701 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381705 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381707 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:24.384622 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381710 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381713 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381715 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381718 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381721 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381724 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381726 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381728 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381731 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381733 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381736 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381739 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381741 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381744 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381747 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381749 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381751 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381754 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381757 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:24.385084 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381759 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381762 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381766 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381770 2568 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381772 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381778 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381780 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381783 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381787 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381790 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381793 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381795 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381798 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381805 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381808 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381811 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381813 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381817 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381819 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:24.385537 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381822 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381824 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381827 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381829 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381832 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381834 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381837 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381840 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381842 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381845 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381847 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381849 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381852 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381854 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381857 2568 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381859 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381862 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381864 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381870 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381872 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:24.385996 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381875 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381877 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381881 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381883 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381886 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381888 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381891 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381893 2568 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381896 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381899 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381901 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381904 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381906 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381909 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381911 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381915 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381919 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381922 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381925 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381927 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:24.386460 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381930 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381932 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381935 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381937 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381940 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381942 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381945 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381947 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381950 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381952 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381956 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381960 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381962 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381965 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381967 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381971 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381974 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381976 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381979 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.381981 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:24.386945 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383245 2568 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383256 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383265 2568 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383269 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383274 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383277 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383281 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383285 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383288 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383291 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383294 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383298 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383301 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383304 2568 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383307 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383310 2568 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383312 2568 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383316 2568 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383320 2568 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383323 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383326 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383329 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383333 2568 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383337 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:27:24.387406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383340 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383343 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383346 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383349 2568 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383352 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383355 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383358 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383361 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383365 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383370 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383373 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383376 2568 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383379 2568 flags.go:64] FLAG: --enable-server="true" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383382 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383387 2568 flags.go:64] FLAG: --event-burst="100" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383390 2568 flags.go:64] FLAG: --event-qps="50" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383393 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383396 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383398 2568 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383403 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383406 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383408 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383411 2568 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383414 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383417 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:27:24.387990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383420 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383423 2568 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383426 2568 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383429 2568 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383432 2568 flags.go:64] FLAG: --feature-gates="" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383436 2568 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383439 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383442 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383445 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383448 2568 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383451 2568 flags.go:64] FLAG: --help="false" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383453 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383457 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383460 2568 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383463 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383466 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383470 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383473 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383476 2568 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383479 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383482 2568 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383485 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383488 2568 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383491 2568 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:27:24.388595 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383493 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383496 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383499 2568 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383501 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383504 2568 flags.go:64] FLAG: --lock-file="" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383507 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383509 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383512 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383517 2568 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383520 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383523 2568 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383526 2568 flags.go:64] FLAG: --logging-format="text" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383529 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383532 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383535 2568 flags.go:64] FLAG: --manifest-url="" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383538 2568 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383542 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383545 2568 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383549 2568 flags.go:64] FLAG: --max-pods="110" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383552 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383555 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383559 2568 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383561 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383564 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383568 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:27:24.389177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383571 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383579 2568 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383582 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383585 2568 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383588 2568 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383591 2568 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383594 2568 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383597 2568 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383600 2568 flags.go:64] FLAG: --port="10250" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383602 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383605 2568 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e6f2ba528be46ad4" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383608 2568 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383611 2568 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383614 2568 flags.go:64] FLAG: --register-node="true" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383617 2568 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383620 2568 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383623 2568 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383626 2568 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383629 2568 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383633 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383636 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383639 2568 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383641 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383644 2568 flags.go:64] FLAG: --runonce="false" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383647 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383650 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383653 2568 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:27:24.389795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383656 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383659 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383662 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383666 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383670 2568 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383673 2568 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383675 2568 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383678 2568 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383681 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383684 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383687 2568 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383701 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383706 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383709 2568 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383712 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383715 2568 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383718 2568 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383721 2568 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383723 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383726 2568 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383729 2568 flags.go:64] FLAG: --v="2" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383734 2568 flags.go:64] FLAG: --version="false" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383738 2568 flags.go:64] FLAG: --vmodule="" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383743 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.383747 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:27:24.390464 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383907 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383911 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383914 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383916 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383919 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383922 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383924 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383927 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383929 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383932 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383935 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383939 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383942 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383944 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383947 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383949 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383951 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383954 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383956 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:24.391050 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383959 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383961 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383963 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383966 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383968 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383971 2568 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383973 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383976 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383979 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383981 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383983 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383987 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383990 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383992 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383995 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.383998 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384000 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384003 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384005 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384008 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:24.391532 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384010 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384012 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384015 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384017 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384021 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384024 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384027 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384029 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384031 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384034 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384036 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384039 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384041 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384044 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384046 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384049 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384051 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384054 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384056 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:24.392016 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384058 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384061 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384063 2568 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384066 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384070 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384073 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384075 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384078 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384080 2568 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384083 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384085 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384088 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384091 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384093 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384097 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384100 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384103 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384106 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384109 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384112 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:24.392459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384114 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384118 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384122 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384125 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384128 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384131 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384133 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384135 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384138 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384140 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384143 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384145 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384147 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384150 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384152 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384155 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384159 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384161 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384164 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:24.392989 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384166 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384169 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384171 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384174 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384176 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384179 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384182 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384184 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384186 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.384212 2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384249 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384253 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384257 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384260 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384262 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384265 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:24.393536 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384267 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384270 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384272 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384275 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384277 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384279 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384282 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384284 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384287 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384289 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384292 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384294 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384297 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384299 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384303 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384305 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384307 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384310 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384312 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384315 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:24.393940 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384317 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384320 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384322 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384325 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384327 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384330 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384332 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384335 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384337 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384346 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384349 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384352 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384354 2568 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384356 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384359 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384361 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384364 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384366 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384369 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384371 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:24.394413 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384374 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384376 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384379 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384381 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384384 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384386 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384390 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384392 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384395 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384397 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384399 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384402 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384405 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384407 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384410 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384412 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384414 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384417 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384419 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384422 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:24.394904 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384425 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384427 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384430 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384433 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384435 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384437 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384440 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384442 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384444 2568 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384447 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384450 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384452 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384455 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384458 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384460 2568 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384464 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384467 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384469 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384473 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:24.395438 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384476 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384479 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384483 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384487 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384490 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384492 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384495 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384497 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384501 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384505 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384507 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384510 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384514 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384516 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384519 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384521 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384524 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384526 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:24.395995 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384529 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384531 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.384534 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.391770 2568 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.391784 2568 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.391800 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.391805 2568 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391859 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391864 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391868 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391871 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391874 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391877 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391880 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391882 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391885 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391888 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391891 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391893 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391896 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:24.396429 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391898 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391901 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391903 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391906 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391908 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391910 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391913 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391916 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391918 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391921 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391924 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391927 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391929 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391932 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391934 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391937 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391940 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391942 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391945 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:24.396913 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391947 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391950 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391952 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391955 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391957 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391960 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391962 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391965 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391967 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391970 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391972 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391975 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391978 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391980 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391983 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391985 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391988 2568 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391990 2568 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391993 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391995 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:24.397358 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.391998 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392000 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392002 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392005 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392008 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392010 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392013 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392015 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392018 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392021 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392024 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392027 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392029 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392032 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392034 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392037 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392039 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392042 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392044 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392047 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:24.397852 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392049 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392051 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392054 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392056 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392059 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392061 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392063 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392066 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392068 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392070 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392073 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392075 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392077 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392080 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392084 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392088 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392092 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392095 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392098 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:24.398368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392101 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392103 2568 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392107 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392110 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392113 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392115 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392118 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392120 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392122 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392125 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392127 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392130 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392133 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392135 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392138 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:24.398838 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.392164 2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392306 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392310 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392313 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392316 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392318 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392321 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392323 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392326 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392329 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392331 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392334 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392336 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392338 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392341 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392343 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392349 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392352 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392354 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:24.399268 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392357 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392359 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392361 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392364 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392366 2568 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392369 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392371 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392374 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392376 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392379 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392381 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392383 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392386 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392389 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392391 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392394 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392396 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392399 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392401 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392404 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:24.399728 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392406 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392409 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392411 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392413 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392417 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392421 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392425 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392428 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392431 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392434 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392437 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392440 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392442 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392445 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392447 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392450 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392452 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392454 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392457 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:24.400366 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392459 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392462 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392465 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392467 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392470 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392472 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392475 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392477 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392480 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392482 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392484 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392487 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392489 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392492 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392494 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392496 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392499 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392501 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392503 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392506 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:24.400887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392508 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392510 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392513 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392515 2568 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392518 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392521 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392523 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392526 2568 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392528 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392531 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392533 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392535 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392538 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392540 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392543 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392545 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392548 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392550 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392552 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392555 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:24.401442 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392557 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392559 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392562 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392564 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392567 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392569 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392571 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392574 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392576 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.392598 2568 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392630 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392633 2568 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392636 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392638 2568 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392641 2568 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:27:24.401925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392643 2568 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392647 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392650 2568 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392653 2568 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392656 2568 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392658 2568 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392661 2568 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392663 2568 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392665 2568 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392668 2568 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392670 2568 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392672 2568 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392675 2568 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392677 2568 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392679 2568 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392682 2568 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392686 2568 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392701 2568 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392704 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392706 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:27:24.402287 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392709 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392712 2568 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392714 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392717 2568 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392719 2568 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392722 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392724 2568 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392727 2568 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392730 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392732 2568 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392735 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392738 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392740 2568 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392744 2568 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392747 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392750 2568 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392752 2568 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392756 2568 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392760 2568 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:27:24.402874 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392763 2568 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392767 2568 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392771 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392773 2568 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392776 2568 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392779 2568 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392781 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392785 2568 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392787 2568 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392792 2568 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392795 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392797 2568 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392800 2568 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392802 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392805 2568 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392807 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392810 2568 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392812 2568 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392815 2568 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392817 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:27:24.403314 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392819 2568 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392822 2568 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392824 2568 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392827 2568 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392829 2568 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392832 2568 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392835 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392838 2568 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392840 2568 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392843 2568 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392845 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392848 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392850 2568 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392852 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392855 2568 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392857 2568 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392860 2568 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392862 2568 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392865 2568 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392868 2568 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:27:24.403820 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392870 2568 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392874 2568 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392876 2568 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392879 2568 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392881 2568 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392884 2568 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392886 2568 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392889 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392892 2568 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392896 2568 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392898 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392901 2568 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392903 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392906 2568 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392908 2568 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392911 2568 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392913 2568 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392916 2568 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392919 2568 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:27:24.404278 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392922 2568 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392924 2568 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:24.392926 2568 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.392975 2568 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.395060 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.395934 2568 server.go:1014] "Starting client certificate rotation" Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.396033 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.396066 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.398462 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:27:24.404736 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.402633 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:27:24.413283 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.413269 2568 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:27:24.416794 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.416780 2568 log.go:25] "Validated CRI v1 image API" Oct 31 00:27:24.417589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.417560 2568 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:27:24.423133 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.423110 2568 fs.go:135] Filesystem UUIDs: map[462dfc59-2b59-4aaa-be6c-d90aeee357e0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe15e7c3-57af-42d9-92d2-22ef711010e7:/dev/nvme0n1p3] Oct 31 00:27:24.423178 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.423134 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:27:24.428380 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.428274 2568 manager.go:217] Machine: {Timestamp:2026-10-31 00:27:24.427028288 +0000 UTC m=+0.367241567 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105358 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f8aefa33f1e6ba02501df9a029d8 SystemUUID:ec25f8ae-fa33-f1e6-ba02-501df9a029d8 BootID:ead03ab6-5c5f-4995-9c3f-8c8102a41b81 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ec:c8:1b:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ec:c8:1b:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:ff:bb:89:4a:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:27:24.428380 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.428379 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:27:24.428492 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.428481 2568 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:27:24.431251 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.431230 2568 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:27:24.431384 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.431253 2568 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-8.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:27:24.431430 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.431386 2568 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:27:24.431430 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.431393 2568 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:27:24.431430 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.431427 2568 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:27:24.431508 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.431446 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:27:24.432475 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.432465 2568 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:27:24.435733 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.435722 2568 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:27:24.435844 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.435835 2568 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:27:24.436636 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.436627 2568 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:27:24.436670 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.436640 2568 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:27:24.436670 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.436651 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:27:24.436670 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.436661 2568 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:27:24.436670 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.436670 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:27:24.436997 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.436986 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:27:24.439346 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.439328 2568 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:27:24.444969 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.444953 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:27:24.445962 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.445943 2568 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:27:24.446002 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.445975 2568 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:27:24.446131 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446121 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:27:24.446167 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446137 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:27:24.446167 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446143 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:27:24.446167 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446148 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:27:24.446167 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446153 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:27:24.446167 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446162 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:27:24.446816 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446808 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:27:24.446851 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446818 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:27:24.446851 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446825 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:27:24.446851 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446831 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:27:24.446851 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446839 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:27:24.446851 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.446849 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:27:24.447598 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.447589 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:27:24.447637 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.447601 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:27:24.451325 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.451313 2568 server.go:1263] "Started kubelet" Oct 31 00:27:24.451411 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.451363 2568 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:27:24.451446 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.451399 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:27:24.451474 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.451464 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:27:24.452011 ip-10-0-129-8 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:27:24.452905 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.452770 2568 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:27:24.459966 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.459948 2568 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:27:24.460358 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.460341 2568 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:27:24.464546 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.464529 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:27:24.465316 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.465300 2568 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:27:24.465420 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.465404 2568 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-8.us-west-2.compute.internal\" not found" Oct 31 00:27:24.465994 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.465976 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:27:24.466103 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.465982 2568 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:27:24.466103 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466012 2568 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:27:24.466190 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466108 2568 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:27:24.466190 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466116 2568 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:27:24.466284 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466262 2568 factory.go:153] Registering CRI-O factory Oct 31 00:27:24.466284 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466281 2568 factory.go:223] Registration of the crio container factory successfully Oct 31 00:27:24.466395 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466323 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:27:24.466395 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466330 2568 factory.go:55] Registering systemd factory Oct 31 00:27:24.466395 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466335 2568 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:27:24.466395 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466347 2568 factory.go:103] Registering Raw factory Oct 31 00:27:24.466395 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.466356 2568 manager.go:1199] Started watching for new ooms in manager Oct 31 00:27:24.467400 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.467385 2568 manager.go:319] Starting recovery of all containers Oct 31 00:27:24.474138 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.474111 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:27:24.477431 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.477407 2568 manager.go:324] Recovery completed Oct 31 00:27:24.481626 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.481614 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:24.483488 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.483470 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:24.483564 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.483502 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:24.483564 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.483513 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:24.484055 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.484043 2568 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:27:24.484108 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.484056 2568 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:27:24.484108 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.484041 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:24.484108 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.484078 2568 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:27:24.484742 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.484725 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:24.484827 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.484758 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:24.484827 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.484775 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:24.485534 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.485523 2568 policy_none.go:50] "Start" Oct 31 00:27:24.485569 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.485538 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:27:24.485569 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.485553 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:27:24.486886 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.486875 2568 policy_none.go:44] "Start" Oct 31 00:27:24.529430 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.529272 2568 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:27:24.535000 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.534985 2568 manager.go:341] "Starting Device Plugin manager" Oct 31 00:27:24.535079 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.535014 2568 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:27:24.535079 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.535023 2568 server.go:87] "Starting device plugin registration server" Oct 31 00:27:24.535290 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.535277 2568 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:27:24.535321 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.535293 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:27:24.535421 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.535400 2568 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:27:24.535526 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.535485 2568 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:27:24.535526 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.535494 2568 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:27:24.536046 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.536025 2568 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:27:24.536119 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.536071 2568 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-8.us-west-2.compute.internal\" not found" Oct 31 00:27:24.588873 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.588824 2568 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-8.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:27:24.590034 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.588826 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-8.us-west-2.compute.internal.18e375a8cb6a4f03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-8.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-8.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:27:24.451286787 +0000 UTC m=+0.391500066,LastTimestamp:2026-10-31 00:27:24.451286787 +0000 UTC m=+0.391500066,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-8.us-west-2.compute.internal,}" Oct 31 00:27:24.605372 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.605353 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:27:24.605372 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.605372 2568 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:27:24.605516 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.605386 2568 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:27:24.605516 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.605419 2568 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:27:24.633823 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.633807 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-8.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:27:24.635887 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.635872 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:24.636610 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.636594 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:24.636682 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.636624 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:24.636682 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.636640 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:24.636682 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.636662 2568 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.676222 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.676198 2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:27:24.676318 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.676261 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bj7wv" Oct 31 00:27:24.706417 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.706393 2568 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal"] Oct 31 00:27:24.706497 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.706471 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:24.709777 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.709763 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:24.709835 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.709789 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:24.709835 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.709804 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:24.711307 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.711289 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:24.711449 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.711436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.711489 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.711462 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:24.712094 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.712078 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:24.712185 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.712110 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:24.712185 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.712118 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:24.712185 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.712131 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:24.712185 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.712136 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:24.712185 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.712149 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:24.713594 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.713579 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.713661 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.713602 2568 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:27:24.714376 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.714359 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:27:24.714470 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.714390 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:27:24.714470 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.714404 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:27:24.732387 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.732371 2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-8.us-west-2.compute.internal\" not found" node="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.735148 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.735133 2568 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-8.us-west-2.compute.internal\" not found" node="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.740333 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.740319 2568 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:24.742198 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.742185 2568 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:24.768325 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.768304 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.768407 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.768328 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.768407 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.768346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaf6c79eda58382b69cd6ec1e1ffc85d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"eaf6c79eda58382b69cd6ec1e1ffc85d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.835234 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.835216 2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:24.836277 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.836264 2568 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:24.836516 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.836505 2568 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.836552 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:24.836522 2568 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-8.us-west-2.compute.internal\": node \"ip-10-0-129-8.us-west-2.compute.internal\" not found" Oct 31 00:27:24.866144 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.866102 2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.869318 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.869300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.869366 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.869331 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.869366 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.869340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.869425 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.869365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaf6c79eda58382b69cd6ec1e1ffc85d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"eaf6c79eda58382b69cd6ec1e1ffc85d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.869425 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.869396 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.869425 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.869404 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaf6c79eda58382b69cd6ec1e1ffc85d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"eaf6c79eda58382b69cd6ec1e1ffc85d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.928127 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.928107 2568 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:27:24.928205 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.928190 2568 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:24.982783 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:24.982771 2568 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:27:25.036153 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.036137 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:25.038826 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.038813 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:27:25.396964 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.396943 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:27:25.397462 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.397085 2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:27:25.397462 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.397079 2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:27:25.397462 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.397087 2568 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:27:25.397462 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.397086 2568 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a5bf11d8d49064afc9ef2bd5f2126127-faee8f219fa21b80.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-8.us-west-2.compute.internal.18e375a8cd55acd6\": read tcp 10.0.129.8:38286->13.58.200.127:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-8.us-west-2.compute.internal.18e375a8cd55acd6 default 5084 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-8.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-8.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-8.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:27:24 +0000 UTC,LastTimestamp:2026-10-31 00:27:24.709778208 +0000 UTC m=+0.649991490,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-8.us-west-2.compute.internal,}" Oct 31 00:27:25.437842 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.437810 2568 apiserver.go:52] "Watching apiserver" Oct 31 00:27:25.465122 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.465107 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:27:25.680739 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.680648 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:22:25 +0000 UTC" deadline="2028-05-18 21:30:54.464790015 +0000 UTC" Oct 31 00:27:25.680739 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.680683 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13581h3m28.784109786s" Oct 31 00:27:25.684823 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.684805 2568 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:27:25.685132 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.685112 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt","openshift-dns/node-resolver-m7t4w","openshift-multus/network-metrics-daemon-ccs74","kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-9dhhl","openshift-image-registry/node-ca-7x6x2","openshift-network-operator/iptables-alerter-68qwm","kube-system/konnectivity-agent-m98mn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-tdb56","openshift-network-diagnostics/network-check-target-gzjf9","openshift-multus/multus-8slwz","openshift-ovn-kubernetes/ovnkube-node-vk9bb","kube-system/global-pull-secret-syncer-ht56p"] Oct 31 00:27:25.686777 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.686759 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.688039 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.688016 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.689376 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.689358 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:25.689445 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.689427 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:25.690741 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.690728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.692007 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.691987 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.693314 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.693297 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.694564 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.694552 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:25.695917 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.695904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.697066 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.697053 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:25.697118 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.697097 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:25.698318 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.698304 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.699917 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.699904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.701271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.701256 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.701339 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.701301 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:25.735538 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.735518 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:27:25.735660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.735643 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:27:25.735750 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.735683 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rlgrr\"" Oct 31 00:27:25.735750 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.735684 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:27:25.736246 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.736232 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:27:25.736776 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.736763 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:27:25.737633 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.737612 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rw8bx\"" Oct 31 00:27:25.739114 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.739092 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:25.739799 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.739265 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:25.739799 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.739383 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cdw8b\"" Oct 31 00:27:25.740257 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.740240 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:27:25.740431 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.740419 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2znq\"" Oct 31 00:27:25.740431 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.740428 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:27:25.740519 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.740504 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:27:25.741748 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.741733 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:27:25.741857 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.741813 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:27:25.742002 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.741930 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jsqnl\"" Oct 31 00:27:25.742066 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.742050 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:27:25.767500 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.767486 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:27:25.775783 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.775761 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-socket-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.775849 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.775783 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-registration-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.775849 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.775800 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-lib-modules\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.775849 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.775815 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qsl4\" (UniqueName: \"kubernetes.io/projected/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-kube-api-access-2qsl4\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.775849 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.775838 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.776036 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.775886 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-socket-dir-parent\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.776036 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.775918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.776036 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.775962 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-run\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.776036 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776014 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/963fd145-6705-467a-bac7-25b3ff8add39-host-slash\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.776180 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776060 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aad04f16-056a-4115-97a0-e75d9e8b36b7-serviceca\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.776180 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-k8s-cni-cncf-io\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.776180 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-bin\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.776180 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776165 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-tmp\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.776324 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.776324 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776249 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1822938-c344-40c9-99c1-986663748169-tmp-dir\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.776324 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776284 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-cni-binary-copy\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.776419 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776321 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-daemon-config\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.776419 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776356 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-systemd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.776419 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776391 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.776419 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776412 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-netd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.776563 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-conf\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.776563 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776456 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.776563 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776504 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.776717 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-systemd-units\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.776717 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776600 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:25.776717 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqq8\" (UniqueName: \"kubernetes.io/projected/963fd145-6705-467a-bac7-25b3ff8add39-kube-api-access-dpqq8\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.776717 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad04f16-056a-4115-97a0-e75d9e8b36b7-host\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.776889 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-kubelet\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.776889 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-conf-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.776889 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-multus-certs\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.776889 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-sys\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.776889 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776844 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-bin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.776889 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776885 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-etc-kubernetes\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.777121 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776911 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-netns\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777121 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776936 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngbqj\" (UniqueName: \"kubernetes.io/projected/b42b8ff9-0140-4b1c-97f2-131bc141352f-kube-api-access-ngbqj\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777121 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.776980 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-host\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.777121 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777014 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.777121 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777060 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-system-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.777121 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-multus\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-kubelet\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-env-overrides\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777160 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trktv\" (UniqueName: \"kubernetes.io/projected/0d861409-3dd5-4eed-ada7-e1817b7fb58f-kube-api-access-trktv\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777175 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-system-cni-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777200 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovn-node-metrics-cert\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-script-lib\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk75x\" (UniqueName: \"kubernetes.io/projected/64a235e0-e510-479f-a6bd-92cc4910a1e5-kube-api-access-qk75x\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777305 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-var-lib-kubelet\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777335 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbng\" (UniqueName: \"kubernetes.io/projected/5476b645-f064-41af-b1da-9f7644f281d0-kube-api-access-chbng\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.777390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777386 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqx6g\" (UniqueName: \"kubernetes.io/projected/b1822938-c344-40c9-99c1-986663748169-kube-api-access-rqx6g\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777422 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-cnibin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777461 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-log-socket\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777497 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-device-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777523 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-slash\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777553 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-hostroot\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777569 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krdc\" (UniqueName: \"kubernetes.io/projected/d6d7d3bf-205d-4251-9b47-400892684e0e-kube-api-access-6krdc\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777604 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777650 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-modprobe-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777683 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysconfig\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777728 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/963fd145-6705-467a-bac7-25b3ff8add39-iptables-alerter-script\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1822938-c344-40c9-99c1-986663748169-hosts-file\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.777806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777834 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-var-lib-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777872 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-ovn\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777948 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-node-log\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-config\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777984 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-kubelet-config\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.777998 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-dbus\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-sys-fs\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778065 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-kubernetes\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778101 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-systemd\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778135 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-agent-certs\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-konnectivity-ca\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-cnibin\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778219 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778257 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-netns\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.778297 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778301 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-etc-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.778802 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.778802 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778355 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-tuned\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.778802 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.778377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6vv\" (UniqueName: \"kubernetes.io/projected/aad04f16-056a-4115-97a0-e75d9e8b36b7-kube-api-access-5z6vv\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.779810 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.779794 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:27:25.779810 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.779801 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:27:25.779944 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.779849 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:27:25.779944 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.779856 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-78fsr\"" Oct 31 00:27:25.779944 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.779888 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:27:25.779944 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.779888 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-46b97\"" Oct 31 00:27:25.834449 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.834429 2568 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:27:25.834552 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.834482 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-66pfr" Oct 31 00:27:25.836181 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836141 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:27:25.836181 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836171 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:27:25.836181 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836171 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:27:25.836382 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836255 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:27:25.836382 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836260 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:27:25.836382 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836269 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:27:25.836517 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836389 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:27:25.836517 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836420 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6gdh\"" Oct 31 00:27:25.836517 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836437 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:27:25.836517 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836433 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:27:25.836517 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836447 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lwt4s\"" Oct 31 00:27:25.836517 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.836488 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:27:25.878895 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.878874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-multus\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.878973 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.878922 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-multus\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.878973 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.878945 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-kubelet\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.878973 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.878959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-env-overrides\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879071 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.878975 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trktv\" (UniqueName: \"kubernetes.io/projected/0d861409-3dd5-4eed-ada7-e1817b7fb58f-kube-api-access-trktv\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:25.879071 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.878993 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-system-cni-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.879071 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879007 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovn-node-metrics-cert\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879071 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-script-lib\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879071 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879092 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-kubelet\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879092 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-system-cni-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879117 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk75x\" (UniqueName: \"kubernetes.io/projected/64a235e0-e510-479f-a6bd-92cc4910a1e5-kube-api-access-qk75x\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-var-lib-kubelet\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879168 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbng\" (UniqueName: \"kubernetes.io/projected/5476b645-f064-41af-b1da-9f7644f281d0-kube-api-access-chbng\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqx6g\" (UniqueName: \"kubernetes.io/projected/b1822938-c344-40c9-99c1-986663748169-kube-api-access-rqx6g\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879212 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879216 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-cnibin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879230 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-var-lib-kubelet\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-log-socket\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879265 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-cnibin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.879271 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-device-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879296 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-slash\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879302 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-log-socket\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-hostroot\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879340 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-slash\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879364 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-device-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879378 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-hostroot\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krdc\" (UniqueName: \"kubernetes.io/projected/d6d7d3bf-205d-4251-9b47-400892684e0e-kube-api-access-6krdc\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879424 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879453 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879478 2568 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879472 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-env-overrides\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-modprobe-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879495 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879509 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysconfig\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/963fd145-6705-467a-bac7-25b3ff8add39-iptables-alerter-script\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1822938-c344-40c9-99c1-986663748169-hosts-file\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.879778 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879578 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysconfig\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879608 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-modprobe-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-var-lib-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879642 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1822938-c344-40c9-99c1-986663748169-hosts-file\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879645 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-ovn\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879681 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-node-log\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879686 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-ovn\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879721 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-config\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879734 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-node-log\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879739 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879748 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-kubelet-config\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879720 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-var-lib-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-dbus\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-sys-fs\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879802 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-kubelet-config\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879817 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-kubernetes\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-systemd\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.880620 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879857 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-dbus\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879853 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-sys-fs\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-agent-certs\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879899 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-systemd\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879906 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-kubernetes\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-konnectivity-ca\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.879949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-cnibin\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880068 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/963fd145-6705-467a-bac7-25b3ff8add39-iptables-alerter-script\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880094 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-cnibin\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880443 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-netns\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880468 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-script-lib\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880504 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-etc-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880472 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-konnectivity-ca\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880519 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-netns\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-etc-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880556 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:25.881438 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880591 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-tuned\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880617 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6vv\" (UniqueName: \"kubernetes.io/projected/aad04f16-056a-4115-97a0-e75d9e8b36b7-kube-api-access-5z6vv\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880645 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-socket-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880668 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-registration-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.880676 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880712 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-lib-modules\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsl4\" (UniqueName: \"kubernetes.io/projected/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-kube-api-access-2qsl4\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880776 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-registration-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880782 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880809 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-socket-dir-parent\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.880879 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret podName:8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:26.380843661 +0000 UTC m=+2.321056934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret") pod "global-pull-secret-syncer-ht56p" (UID: "8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880888 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.880946 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-socket-dir-parent\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-run\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/963fd145-6705-467a-bac7-25b3ff8add39-host-slash\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aad04f16-056a-4115-97a0-e75d9e8b36b7-serviceca\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.882220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-k8s-cni-cncf-io\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-bin\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-tmp\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881188 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-run\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881131 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-lib-modules\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881210 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1822938-c344-40c9-99c1-986663748169-tmp-dir\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-cni-binary-copy\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-daemon-config\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881327 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/963fd145-6705-467a-bac7-25b3ff8add39-host-slash\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-systemd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881367 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-bin\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881410 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-netd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881417 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-k8s-cni-cncf-io\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-conf\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881459 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881471 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-config\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883093 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881483 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881508 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-systemd-units\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881533 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881557 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqq8\" (UniqueName: \"kubernetes.io/projected/963fd145-6705-467a-bac7-25b3ff8add39-kube-api-access-dpqq8\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad04f16-056a-4115-97a0-e75d9e8b36b7-host\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881607 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-kubelet\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881632 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-conf-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881658 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-multus-certs\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881679 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-sys\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-bin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881749 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-etc-kubernetes\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881758 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1822938-c344-40c9-99c1-986663748169-tmp-dir\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-netns\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbqj\" (UniqueName: \"kubernetes.io/projected/b42b8ff9-0140-4b1c-97f2-131bc141352f-kube-api-access-ngbqj\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-host\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881830 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-system-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881892 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aad04f16-056a-4115-97a0-e75d9e8b36b7-serviceca\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.883589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881936 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-system-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881373 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-systemd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881985 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-netd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.881410 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882051 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-cni-binary-copy\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882111 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-conf\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882112 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-conf-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882169 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-multus-certs\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-sys\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882245 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882250 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-daemon-config\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882280 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-bin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882326 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-host\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882348 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882394 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-systemd-units\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882454 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-kubelet\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882494 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-etc-kubernetes\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882500 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad04f16-056a-4115-97a0-e75d9e8b36b7-host\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882547 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-netns\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.884129 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.882567 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.882627 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs podName:0d861409-3dd5-4eed-ada7-e1817b7fb58f nodeName:}" failed. No retries permitted until 2026-10-31 00:27:26.382612586 +0000 UTC m=+2.322825867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs") pod "network-metrics-daemon-ccs74" (UID: "0d861409-3dd5-4eed-ada7-e1817b7fb58f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.882738 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-socket-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.883102 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-tuned\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.883223 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovn-node-metrics-cert\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.883832 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.884097 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-agent-certs\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.884141 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.884165 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.884624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.884279 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-tmp\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.938319 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.938266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trktv\" (UniqueName: \"kubernetes.io/projected/0d861409-3dd5-4eed-ada7-e1817b7fb58f-kube-api-access-trktv\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:25.938500 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.938483 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbng\" (UniqueName: \"kubernetes.io/projected/5476b645-f064-41af-b1da-9f7644f281d0-kube-api-access-chbng\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:25.939667 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.939650 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krdc\" (UniqueName: \"kubernetes.io/projected/d6d7d3bf-205d-4251-9b47-400892684e0e-kube-api-access-6krdc\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:27:25.940522 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.940505 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6vv\" (UniqueName: \"kubernetes.io/projected/aad04f16-056a-4115-97a0-e75d9e8b36b7-kube-api-access-5z6vv\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:25.943491 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.943470 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:25.943626 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.943498 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:25.943626 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.943510 2568 projected.go:196] Error preparing data for projected volume kube-api-access-f87bx for pod openshift-network-diagnostics/network-check-target-gzjf9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:25.943626 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:25.943572 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx podName:8b9dc277-9345-44d1-9f7e-90b5a9f3380e nodeName:}" failed. No retries permitted until 2026-10-31 00:27:26.443556394 +0000 UTC m=+2.383769685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f87bx" (UniqueName: "kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx") pod "network-check-target-gzjf9" (UID: "8b9dc277-9345-44d1-9f7e-90b5a9f3380e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:25.944340 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.944314 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqx6g\" (UniqueName: \"kubernetes.io/projected/b1822938-c344-40c9-99c1-986663748169-kube-api-access-rqx6g\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:25.944600 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.944463 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsl4\" (UniqueName: \"kubernetes.io/projected/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-kube-api-access-2qsl4\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:25.944948 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.944927 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk75x\" (UniqueName: \"kubernetes.io/projected/64a235e0-e510-479f-a6bd-92cc4910a1e5-kube-api-access-qk75x\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:25.945255 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.945233 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqq8\" (UniqueName: \"kubernetes.io/projected/963fd145-6705-467a-bac7-25b3ff8add39-kube-api-access-dpqq8\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:25.946010 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:25.945992 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbqj\" (UniqueName: \"kubernetes.io/projected/b42b8ff9-0140-4b1c-97f2-131bc141352f-kube-api-access-ngbqj\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:26.010476 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.010449 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:27:26.025335 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.025316 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:27:26.041516 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.041498 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:27:26.058785 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.058764 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:27:26.060476 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.060462 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:27:26.077385 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.077364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:26.090007 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.089989 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:27:26.095914 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.095896 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8slwz" Oct 31 00:27:26.109420 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.109397 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:27:26.209386 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:26.209357 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada14adc9fde4e31f8329dced7d0e766.slice/crio-4ddd7ba8d34be7c0899e8c77bb5d0bd116657b2b0bf891271a6147ae1fbe39a8 WatchSource:0}: Error finding container 4ddd7ba8d34be7c0899e8c77bb5d0bd116657b2b0bf891271a6147ae1fbe39a8: Status 404 returned error can't find the container with id 4ddd7ba8d34be7c0899e8c77bb5d0bd116657b2b0bf891271a6147ae1fbe39a8 Oct 31 00:27:26.209925 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:26.209902 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf6c79eda58382b69cd6ec1e1ffc85d.slice/crio-e059d0603b0e71ebf91d721d4911a8d57b734f52c1589e2e179a8aef6b367802 WatchSource:0}: Error finding container e059d0603b0e71ebf91d721d4911a8d57b734f52c1589e2e179a8aef6b367802: Status 404 returned error can't find the container with id e059d0603b0e71ebf91d721d4911a8d57b734f52c1589e2e179a8aef6b367802 Oct 31 00:27:26.215705 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.215690 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:27:26.385339 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.385302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:26.385483 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.385358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:26.385483 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:26.385442 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:26.385483 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:26.385446 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:26.385607 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:26.385503 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs podName:0d861409-3dd5-4eed-ada7-e1817b7fb58f nodeName:}" failed. No retries permitted until 2026-10-31 00:27:27.385487753 +0000 UTC m=+3.325701021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs") pod "network-metrics-daemon-ccs74" (UID: "0d861409-3dd5-4eed-ada7-e1817b7fb58f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:26.385607 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:26.385518 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret podName:8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:27.385511666 +0000 UTC m=+3.325724934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret") pod "global-pull-secret-syncer-ht56p" (UID: "8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:26.486122 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.486051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:26.486635 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:26.486161 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:26.486635 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:26.486197 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:26.486635 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:26.486214 2568 projected.go:196] Error preparing data for projected volume kube-api-access-f87bx for pod openshift-network-diagnostics/network-check-target-gzjf9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:26.486635 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:26.486258 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx podName:8b9dc277-9345-44d1-9f7e-90b5a9f3380e nodeName:}" failed. No retries permitted until 2026-10-31 00:27:27.486245776 +0000 UTC m=+3.426459043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f87bx" (UniqueName: "kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx") pod "network-check-target-gzjf9" (UID: "8b9dc277-9345-44d1-9f7e-90b5a9f3380e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:26.572966 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.572942 2568 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:26.609566 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.609521 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"ada14adc9fde4e31f8329dced7d0e766","Type":"ContainerStarted","Data":"4ddd7ba8d34be7c0899e8c77bb5d0bd116657b2b0bf891271a6147ae1fbe39a8"} Oct 31 00:27:26.610674 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.610653 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"eaf6c79eda58382b69cd6ec1e1ffc85d","Type":"ContainerStarted","Data":"e059d0603b0e71ebf91d721d4911a8d57b734f52c1589e2e179a8aef6b367802"} Oct 31 00:27:26.836685 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.836576 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:22:26 +0000 UTC" deadline="2028-05-15 02:37:04.664953949 +0000 UTC" Oct 31 00:27:26.836685 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.836623 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13490h9m37.828335798s" Oct 31 00:27:26.865796 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:26.865765 2568 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:26.952149 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:26.952117 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1822938_c344_40c9_99c1_986663748169.slice/crio-957b2fe6a5227d581b085f387356ffaf5ac37ba89d6d5abe6fe6eb147c39db86 WatchSource:0}: Error finding container 957b2fe6a5227d581b085f387356ffaf5ac37ba89d6d5abe6fe6eb147c39db86: Status 404 returned error can't find the container with id 957b2fe6a5227d581b085f387356ffaf5ac37ba89d6d5abe6fe6eb147c39db86 Oct 31 00:27:26.969414 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:26.969392 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a235e0_e510_479f_a6bd_92cc4910a1e5.slice/crio-dc90f34210244e8bc0b66d0db9eaf95da4c4de912a114d9ff3702d4854014d65 WatchSource:0}: Error finding container dc90f34210244e8bc0b66d0db9eaf95da4c4de912a114d9ff3702d4854014d65: Status 404 returned error can't find the container with id dc90f34210244e8bc0b66d0db9eaf95da4c4de912a114d9ff3702d4854014d65 Oct 31 00:27:26.996887 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:26.996863 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75cde1b_aaee_46c8_b956_dcecf3b6afbf.slice/crio-93567fb71b263cb3b775692c7c043e352bf3e90c5255a676ac23d752ca02ecc6 WatchSource:0}: Error finding container 93567fb71b263cb3b775692c7c043e352bf3e90c5255a676ac23d752ca02ecc6: Status 404 returned error can't find the container with id 93567fb71b263cb3b775692c7c043e352bf3e90c5255a676ac23d752ca02ecc6 Oct 31 00:27:27.008921 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:27.008900 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963fd145_6705_467a_bac7_25b3ff8add39.slice/crio-1d02da165f9aba2e1411d24cfc79250d0608eacb72f3fb742edc1e6ddb9fe517 WatchSource:0}: Error finding container 1d02da165f9aba2e1411d24cfc79250d0608eacb72f3fb742edc1e6ddb9fe517: Status 404 returned error can't find the container with id 1d02da165f9aba2e1411d24cfc79250d0608eacb72f3fb742edc1e6ddb9fe517 Oct 31 00:27:27.029098 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:27.029073 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42b8ff9_0140_4b1c_97f2_131bc141352f.slice/crio-f3d8a2f581559afaed0caeb6c409eeaf66a72fd367483fa96e94a6649fadb40b WatchSource:0}: Error finding container f3d8a2f581559afaed0caeb6c409eeaf66a72fd367483fa96e94a6649fadb40b: Status 404 returned error can't find the container with id f3d8a2f581559afaed0caeb6c409eeaf66a72fd367483fa96e94a6649fadb40b Oct 31 00:27:27.036807 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:27.036787 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5476b645_f064_41af_b1da_9f7644f281d0.slice/crio-74e42a6ef1d8accc3c3df3bbe9c8a6c41da448a0619a1933337b9d406b0e1844 WatchSource:0}: Error finding container 74e42a6ef1d8accc3c3df3bbe9c8a6c41da448a0619a1933337b9d406b0e1844: Status 404 returned error can't find the container with id 74e42a6ef1d8accc3c3df3bbe9c8a6c41da448a0619a1933337b9d406b0e1844 Oct 31 00:27:27.040795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.039073 2568 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:27:27.110351 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:27.110324 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d7d3bf_205d_4251_9b47_400892684e0e.slice/crio-5bf5130065b585a73bd9710561277316c817d70e2932411408d85f8863430b3c WatchSource:0}: Error finding container 5bf5130065b585a73bd9710561277316c817d70e2932411408d85f8863430b3c: Status 404 returned error can't find the container with id 5bf5130065b585a73bd9710561277316c817d70e2932411408d85f8863430b3c Oct 31 00:27:27.135191 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:27.135167 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad04f16_056a_4115_97a0_e75d9e8b36b7.slice/crio-325df31cc31fd7f9e658af6a13c49330aecddac9cd4f4b550230fe96f2f0a7af WatchSource:0}: Error finding container 325df31cc31fd7f9e658af6a13c49330aecddac9cd4f4b550230fe96f2f0a7af: Status 404 returned error can't find the container with id 325df31cc31fd7f9e658af6a13c49330aecddac9cd4f4b550230fe96f2f0a7af Oct 31 00:27:27.173368 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:27:27.173341 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0284bc6_6a9b_42ed_b443_1f94367c8ac4.slice/crio-58be41a2e2343a7d482c943cc79b4e39ccf6cb6b19b905a24e688f01d883720e WatchSource:0}: Error finding container 58be41a2e2343a7d482c943cc79b4e39ccf6cb6b19b905a24e688f01d883720e: Status 404 returned error can't find the container with id 58be41a2e2343a7d482c943cc79b4e39ccf6cb6b19b905a24e688f01d883720e Oct 31 00:27:27.394262 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.394171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:27.394262 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.394222 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:27.394520 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.394350 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:27.394520 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.394352 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:27.394520 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.394422 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs podName:0d861409-3dd5-4eed-ada7-e1817b7fb58f nodeName:}" failed. No retries permitted until 2026-10-31 00:27:29.394401984 +0000 UTC m=+5.334615270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs") pod "network-metrics-daemon-ccs74" (UID: "0d861409-3dd5-4eed-ada7-e1817b7fb58f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:27.394520 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.394442 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret podName:8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:29.394431785 +0000 UTC m=+5.334645061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret") pod "global-pull-secret-syncer-ht56p" (UID: "8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:27.494978 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.494942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:27.495610 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.495140 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:27.495610 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.495166 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:27.495610 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.495179 2568 projected.go:196] Error preparing data for projected volume kube-api-access-f87bx for pod openshift-network-diagnostics/network-check-target-gzjf9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:27.495610 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.495246 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx podName:8b9dc277-9345-44d1-9f7e-90b5a9f3380e nodeName:}" failed. No retries permitted until 2026-10-31 00:27:29.495225606 +0000 UTC m=+5.435438880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f87bx" (UniqueName: "kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx") pod "network-check-target-gzjf9" (UID: "8b9dc277-9345-44d1-9f7e-90b5a9f3380e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:27.606406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.606372 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:27.606551 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.606443 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:27.606551 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.606540 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:27.606708 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.606540 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:27.606708 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.606619 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:27.606708 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:27.606674 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:27.614324 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.614299 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" event={"ID":"d0284bc6-6a9b-42ed-b443-1f94367c8ac4","Type":"ContainerStarted","Data":"58be41a2e2343a7d482c943cc79b4e39ccf6cb6b19b905a24e688f01d883720e"} Oct 31 00:27:27.616046 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.616021 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7x6x2" event={"ID":"aad04f16-056a-4115-97a0-e75d9e8b36b7","Type":"ContainerStarted","Data":"325df31cc31fd7f9e658af6a13c49330aecddac9cd4f4b550230fe96f2f0a7af"} Oct 31 00:27:27.618016 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.617993 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8slwz" event={"ID":"d6d7d3bf-205d-4251-9b47-400892684e0e","Type":"ContainerStarted","Data":"5bf5130065b585a73bd9710561277316c817d70e2932411408d85f8863430b3c"} Oct 31 00:27:27.619483 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.619459 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-68qwm" event={"ID":"963fd145-6705-467a-bac7-25b3ff8add39","Type":"ContainerStarted","Data":"1d02da165f9aba2e1411d24cfc79250d0608eacb72f3fb742edc1e6ddb9fe517"} Oct 31 00:27:27.620878 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.620846 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m98mn" event={"ID":"b75cde1b-aaee-46c8-b956-dcecf3b6afbf","Type":"ContainerStarted","Data":"93567fb71b263cb3b775692c7c043e352bf3e90c5255a676ac23d752ca02ecc6"} Oct 31 00:27:27.622332 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.622308 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" event={"ID":"64a235e0-e510-479f-a6bd-92cc4910a1e5","Type":"ContainerStarted","Data":"dc90f34210244e8bc0b66d0db9eaf95da4c4de912a114d9ff3702d4854014d65"} Oct 31 00:27:27.624165 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.624141 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerStarted","Data":"74e42a6ef1d8accc3c3df3bbe9c8a6c41da448a0619a1933337b9d406b0e1844"} Oct 31 00:27:27.625813 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.625784 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"f3d8a2f581559afaed0caeb6c409eeaf66a72fd367483fa96e94a6649fadb40b"} Oct 31 00:27:27.627082 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.627061 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m7t4w" event={"ID":"b1822938-c344-40c9-99c1-986663748169","Type":"ContainerStarted","Data":"957b2fe6a5227d581b085f387356ffaf5ac37ba89d6d5abe6fe6eb147c39db86"} Oct 31 00:27:27.837058 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.836952 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:22:26 +0000 UTC" deadline="2028-05-14 14:50:03.389083127 +0000 UTC" Oct 31 00:27:27.837058 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:27.837005 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13478h22m35.552081971s" Oct 31 00:27:29.411691 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:29.411590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:29.411691 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:29.411687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:29.412445 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.411911 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:29.412445 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.411979 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret podName:8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:33.41195989 +0000 UTC m=+9.352173162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret") pod "global-pull-secret-syncer-ht56p" (UID: "8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:29.412445 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.412054 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:29.412445 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.412085 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs podName:0d861409-3dd5-4eed-ada7-e1817b7fb58f nodeName:}" failed. No retries permitted until 2026-10-31 00:27:33.412076581 +0000 UTC m=+9.352289849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs") pod "network-metrics-daemon-ccs74" (UID: "0d861409-3dd5-4eed-ada7-e1817b7fb58f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:29.512667 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:29.512316 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:29.512667 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.512539 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:29.512667 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.512562 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:29.512667 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.512576 2568 projected.go:196] Error preparing data for projected volume kube-api-access-f87bx for pod openshift-network-diagnostics/network-check-target-gzjf9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:29.512667 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.512631 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx podName:8b9dc277-9345-44d1-9f7e-90b5a9f3380e nodeName:}" failed. No retries permitted until 2026-10-31 00:27:33.512612749 +0000 UTC m=+9.452826016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f87bx" (UniqueName: "kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx") pod "network-check-target-gzjf9" (UID: "8b9dc277-9345-44d1-9f7e-90b5a9f3380e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:29.606357 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:29.606321 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:29.606540 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.606478 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:29.606617 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:29.606543 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:29.606679 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.606642 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:29.606732 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:29.606688 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:29.606834 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:29.606796 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:30.670467 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:30.670424 2568 generic.go:356] "Generic (PLEG): container finished" podID="ada14adc9fde4e31f8329dced7d0e766" containerID="cdd5cbd178dbeeb2ec0ef1150f777543f29d39fde2416ddf0a65669eebfbed3f" exitCode=0 Oct 31 00:27:30.670972 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:30.670529 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"ada14adc9fde4e31f8329dced7d0e766","Type":"ContainerDied","Data":"cdd5cbd178dbeeb2ec0ef1150f777543f29d39fde2416ddf0a65669eebfbed3f"} Oct 31 00:27:30.681108 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:30.681042 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"eaf6c79eda58382b69cd6ec1e1ffc85d","Type":"ContainerStarted","Data":"4f0ebf687d99c395d11016ed38d9938dea8ac8749902332c9853e36138740936"} Oct 31 00:27:31.606459 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:31.606423 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:31.606654 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:31.606561 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:31.606654 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:31.606648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:31.606814 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:31.606763 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:31.606876 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:31.606829 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:31.606932 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:31.606906 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:31.687604 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:31.687564 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"ada14adc9fde4e31f8329dced7d0e766","Type":"ContainerStarted","Data":"35dbcec67c9c79ac3646c512b5d2065d67729b71a3275fa89d806e30bf13359d"} Oct 31 00:27:31.694947 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:31.694908 2568 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="b26ca43ba5ad9171c768c288bd6a8ab593eb06e793388bc4af524069198b7d57" exitCode=0 Oct 31 00:27:31.696083 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:31.696027 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerDied","Data":"b26ca43ba5ad9171c768c288bd6a8ab593eb06e793388bc4af524069198b7d57"} Oct 31 00:27:31.808185 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:31.808113 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" podStartSLOduration=6.808089935 podStartE2EDuration="6.808089935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:30.906943434 +0000 UTC m=+6.847156756" watchObservedRunningTime="2026-10-31 00:27:31.808089935 +0000 UTC m=+7.748303231" Oct 31 00:27:31.933007 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:31.932950 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" podStartSLOduration=6.932935822 podStartE2EDuration="6.932935822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:31.808535107 +0000 UTC m=+7.748748401" watchObservedRunningTime="2026-10-31 00:27:31.932935822 +0000 UTC m=+7.873149148" Oct 31 00:27:32.708992 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:32.708840 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m98mn" event={"ID":"b75cde1b-aaee-46c8-b956-dcecf3b6afbf","Type":"ContainerStarted","Data":"257c21cc8d15e712c0392d7da2ddf35dd95a9df920b3fd81fe8879ef6ed12ba8"} Oct 31 00:27:33.445176 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:33.445087 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:33.445176 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:33.445165 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:33.445443 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.445330 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:33.445443 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.445402 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs podName:0d861409-3dd5-4eed-ada7-e1817b7fb58f nodeName:}" failed. No retries permitted until 2026-10-31 00:27:41.445380884 +0000 UTC m=+17.385594156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs") pod "network-metrics-daemon-ccs74" (UID: "0d861409-3dd5-4eed-ada7-e1817b7fb58f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:33.445907 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.445870 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:33.446034 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.445931 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret podName:8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:41.445913534 +0000 UTC m=+17.386126817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret") pod "global-pull-secret-syncer-ht56p" (UID: "8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:33.546366 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:33.546324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:33.546546 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.546511 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:33.546546 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.546532 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:33.546546 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.546546 2568 projected.go:196] Error preparing data for projected volume kube-api-access-f87bx for pod openshift-network-diagnostics/network-check-target-gzjf9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:33.546677 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.546606 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx podName:8b9dc277-9345-44d1-9f7e-90b5a9f3380e nodeName:}" failed. No retries permitted until 2026-10-31 00:27:41.546589058 +0000 UTC m=+17.486802342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f87bx" (UniqueName: "kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx") pod "network-check-target-gzjf9" (UID: "8b9dc277-9345-44d1-9f7e-90b5a9f3380e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:33.606379 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:33.606341 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:33.606379 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:33.606367 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:33.606634 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.606508 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:33.606634 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.606561 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:33.606634 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:33.606583 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:33.606826 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:33.606667 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:33.707236 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:33.707154 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7x6x2" event={"ID":"aad04f16-056a-4115-97a0-e75d9e8b36b7","Type":"ContainerStarted","Data":"50f9ff65d817bb21f85cee52c253db76ec48e46446c44b30cfd37a24ae8ecf32"} Oct 31 00:27:33.826230 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:33.826169 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m98mn" podStartSLOduration=4.012876615 podStartE2EDuration="8.826146362s" totalImagesPullingTime="4.813269747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:32.831108511 +0000 UTC m=+8.771321805" watchObservedRunningTime="2026-10-31 00:27:33.826146362 +0000 UTC m=+9.766359651" Oct 31 00:27:34.716317 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:34.716276 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" event={"ID":"64a235e0-e510-479f-a6bd-92cc4910a1e5","Type":"ContainerStarted","Data":"d24a1b55dc12e61986657c73fc4b00cc8e5194ab21d7b97c3d0e21f1b8a63d46"} Oct 31 00:27:35.038428 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:35.037498 2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:35.039429 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:35.039397 2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:35.171076 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:35.170174 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7x6x2" podStartSLOduration=4.254992577 podStartE2EDuration="10.170153177s" totalImagesPullingTime="5.9151606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:33.827569234 +0000 UTC m=+9.767782528" watchObservedRunningTime="2026-10-31 00:27:35.170153177 +0000 UTC m=+11.110366469" Oct 31 00:27:35.606373 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:35.606336 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:35.606533 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:35.606504 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:35.606951 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:35.606932 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:35.607082 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:35.607053 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:35.607134 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:35.607121 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:35.607227 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:35.607208 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:36.161288 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.161239 2568 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:27:36.543168 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.543062 2568 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:27:36.161262366Z","UUID":"3b179745-f4e2-4783-a6fa-3dedfaad1388","Handler":null,"Name":"","Endpoint":""} Oct 31 00:27:36.545068 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.545009 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:27:36.545068 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.545049 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:27:36.722441 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.722399 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8slwz" event={"ID":"d6d7d3bf-205d-4251-9b47-400892684e0e","Type":"ContainerStarted","Data":"f2bb2055265f993662fe953759082ca750a08605b7a67db36304770cbd48cb4d"} Oct 31 00:27:36.725879 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.725846 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" event={"ID":"64a235e0-e510-479f-a6bd-92cc4910a1e5","Type":"ContainerStarted","Data":"2d753bbfcfec8cca55d08ef1237821ce9d6c257aa30f3f901249ca2435522e40"} Oct 31 00:27:36.728451 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.728423 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerStarted","Data":"1d5274c8a659aaf6fbcbd518031517f8545c507beabf9dc576c7565fcc7d5266"} Oct 31 00:27:36.730133 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.730104 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m7t4w" event={"ID":"b1822938-c344-40c9-99c1-986663748169","Type":"ContainerStarted","Data":"afe7ed34efe3993fabcaf00d4169d30ef93a77ca401d764567824ad167e20898"} Oct 31 00:27:36.961191 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.960176 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-m7t4w" podStartSLOduration=2.813196087 podStartE2EDuration="11.960157139s" totalImagesPullingTime="9.146961052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:36.95986865 +0000 UTC m=+12.900081942" watchObservedRunningTime="2026-10-31 00:27:36.960157139 +0000 UTC m=+12.900370432" Oct 31 00:27:36.961191 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:36.960950 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8slwz" podStartSLOduration=3.284390635 podStartE2EDuration="11.960937235s" totalImagesPullingTime="8.6765466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:36.843676062 +0000 UTC m=+12.783889353" watchObservedRunningTime="2026-10-31 00:27:36.960937235 +0000 UTC m=+12.901150528" Oct 31 00:27:37.606451 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:37.606417 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:37.606970 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:37.606417 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:37.606970 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:37.606549 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:37.606970 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:37.606435 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:37.606970 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:37.606674 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:37.606970 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:37.606790 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:37.734738 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:37.734637 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-68qwm" event={"ID":"963fd145-6705-467a-bac7-25b3ff8add39","Type":"ContainerStarted","Data":"f7f5f89f4ce545265c997b544fad362995a2305528492aa33f71d86c388824c1"} Oct 31 00:27:37.737505 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:37.737476 2568 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="1d5274c8a659aaf6fbcbd518031517f8545c507beabf9dc576c7565fcc7d5266" exitCode=0 Oct 31 00:27:37.738542 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:37.738518 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerDied","Data":"1d5274c8a659aaf6fbcbd518031517f8545c507beabf9dc576c7565fcc7d5266"} Oct 31 00:27:37.851843 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:37.851796 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-68qwm" podStartSLOduration=3.962614909 podStartE2EDuration="12.851763351s" totalImagesPullingTime="8.889148442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:37.851484524 +0000 UTC m=+13.791697815" watchObservedRunningTime="2026-10-31 00:27:37.851763351 +0000 UTC m=+13.791976643" Oct 31 00:27:38.740779 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:38.740743 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" event={"ID":"d0284bc6-6a9b-42ed-b443-1f94367c8ac4","Type":"ContainerStarted","Data":"7f3a34a118824552556aa12e4e6b66999a10aaf6a2d02e1be7b7ff084d98f20f"} Oct 31 00:27:38.742673 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:38.742649 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" event={"ID":"64a235e0-e510-479f-a6bd-92cc4910a1e5","Type":"ContainerStarted","Data":"e642fd096d9b332ca5e4cc90cf8a063e1d1dea874cd45bde3d9ce5d618702399"} Oct 31 00:27:38.850729 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:38.850664 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" podStartSLOduration=3.093865791 podStartE2EDuration="13.850653065s" totalImagesPullingTime="10.756787274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:38.849955487 +0000 UTC m=+14.790168775" watchObservedRunningTime="2026-10-31 00:27:38.850653065 +0000 UTC m=+14.790866349" Oct 31 00:27:38.962258 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:38.962217 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" podStartSLOduration=3.434287307 podStartE2EDuration="13.962205763s" totalImagesPullingTime="10.527918456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:38.962150931 +0000 UTC m=+14.902364219" watchObservedRunningTime="2026-10-31 00:27:38.962205763 +0000 UTC m=+14.902419045" Oct 31 00:27:39.606111 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:39.606084 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:39.606241 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:39.606118 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:39.606241 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:39.606196 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:39.606241 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:39.606202 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:39.606342 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:39.606270 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:39.606372 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:39.606359 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:39.687415 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:39.687389 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:39.687526 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:39.687507 2568 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:27:39.687910 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:39.687891 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:27:39.746798 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:39.746775 2568 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="a9c4bc84c620042da09dfb88c7c12aa88d0f50aee414ef1aac74c619dc710f4d" exitCode=0 Oct 31 00:27:39.747377 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:39.746801 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerDied","Data":"a9c4bc84c620042da09dfb88c7c12aa88d0f50aee414ef1aac74c619dc710f4d"} Oct 31 00:27:41.513206 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:41.512985 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:41.513206 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:41.513183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:41.513550 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.513127 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:41.513550 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.513282 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs podName:0d861409-3dd5-4eed-ada7-e1817b7fb58f nodeName:}" failed. No retries permitted until 2026-10-31 00:27:57.513263019 +0000 UTC m=+33.453476306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs") pod "network-metrics-daemon-ccs74" (UID: "0d861409-3dd5-4eed-ada7-e1817b7fb58f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:41.513550 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.513287 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:41.513550 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.513322 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret podName:8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5 nodeName:}" failed. No retries permitted until 2026-10-31 00:27:57.513312694 +0000 UTC m=+33.453525973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret") pod "global-pull-secret-syncer-ht56p" (UID: "8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:41.605812 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:41.605786 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:41.605923 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:41.605900 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:41.606011 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.605910 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:41.606011 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:41.605927 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:41.606011 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.605992 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:41.606109 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.606049 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:41.613925 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:41.613910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:41.614034 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.614023 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:41.614074 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.614039 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:41.614074 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.614047 2568 projected.go:196] Error preparing data for projected volume kube-api-access-f87bx for pod openshift-network-diagnostics/network-check-target-gzjf9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:41.614137 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:41.614080 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx podName:8b9dc277-9345-44d1-9f7e-90b5a9f3380e nodeName:}" failed. No retries permitted until 2026-10-31 00:27:57.614070224 +0000 UTC m=+33.554283490 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f87bx" (UniqueName: "kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx") pod "network-check-target-gzjf9" (UID: "8b9dc277-9345-44d1-9f7e-90b5a9f3380e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:41.752957 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:41.752935 2568 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="042b672e24a45f9b98a0f0c55a101333dd651a25ad2d9584c05be1d7f82a0b59" exitCode=0 Oct 31 00:27:41.753086 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:41.752979 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerDied","Data":"042b672e24a45f9b98a0f0c55a101333dd651a25ad2d9584c05be1d7f82a0b59"} Oct 31 00:27:43.606502 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:43.606467 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:43.606502 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:43.606485 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:43.606996 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:43.606588 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:43.606996 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:43.606626 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:43.606996 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:43.606680 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:43.606996 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:43.606763 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:44.761538 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:44.761512 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerStarted","Data":"a9d5ec98d3e248f651fcde3acd51ffda93adfa6ca99d28ccd14268237f4d9d75"} Oct 31 00:27:45.605572 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:45.605543 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:45.605787 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:45.605543 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:45.605787 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:45.605657 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:45.605787 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:45.605543 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:45.605787 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:45.605715 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:45.605996 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:45.605787 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:45.765667 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:45.765643 2568 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="a9d5ec98d3e248f651fcde3acd51ffda93adfa6ca99d28ccd14268237f4d9d75" exitCode=0 Oct 31 00:27:45.766001 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:45.765675 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerDied","Data":"a9d5ec98d3e248f651fcde3acd51ffda93adfa6ca99d28ccd14268237f4d9d75"} Oct 31 00:27:46.770240 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:46.770210 2568 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="7856a5b9e384a2f5cc944ccba6518e1b4323906c20fcb8d5e83aa6ed2085fb91" exitCode=0 Oct 31 00:27:46.770724 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:46.770255 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerDied","Data":"7856a5b9e384a2f5cc944ccba6518e1b4323906c20fcb8d5e83aa6ed2085fb91"} Oct 31 00:27:47.606172 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:47.606142 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:47.606172 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:47.606179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:47.606404 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:47.606176 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:47.606404 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:47.606275 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:47.606404 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:47.606317 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:47.606404 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:47.606379 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:47.775739 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:47.775716 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tdb56" event={"ID":"5476b645-f064-41af-b1da-9f7644f281d0","Type":"ContainerStarted","Data":"1e0ddd6432fe47ac826f9b76adfb03fa2b0138da008893c0a79901fca701d05e"} Oct 31 00:27:47.983321 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:47.983221 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tdb56" podStartSLOduration=9.024927035 podStartE2EDuration="22.983209187s" totalImagesPullingTime="13.958282152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:47.982611577 +0000 UTC m=+23.922824865" watchObservedRunningTime="2026-10-31 00:27:47.983209187 +0000 UTC m=+23.923422472" Oct 31 00:27:49.605827 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:49.605797 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:49.606289 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:49.605797 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:49.606289 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:49.605920 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:49.606289 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:49.605922 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:49.606289 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:49.605994 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:49.606289 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:49.606052 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:51.606600 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:51.606567 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:51.606600 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:51.606588 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:51.607038 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:51.606573 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:51.607038 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:51.606711 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:51.607038 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:51.606813 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:51.607038 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:51.606893 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:53.605673 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:53.605648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:53.606107 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:53.605648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:53.606107 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:53.605766 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:53.606107 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:53.605650 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:53.606107 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:53.605830 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:53.606107 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:53.605887 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:55.605984 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:55.605949 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:55.606454 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:55.606014 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:55.606454 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:55.606112 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:55.606454 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:55.606118 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:55.606454 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:55.606224 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:55.606454 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:55.606284 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:57.443182 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:57.443155 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m7t4w_b1822938-c344-40c9-99c1-986663748169/dns-node-resolver/0.log" Oct 31 00:27:57.528173 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:57.528136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:57.528173 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:57.528181 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:57.528382 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.528290 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:57.528382 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.528340 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs podName:0d861409-3dd5-4eed-ada7-e1817b7fb58f nodeName:}" failed. No retries permitted until 2026-10-31 00:28:29.528327508 +0000 UTC m=+65.468540774 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs") pod "network-metrics-daemon-ccs74" (UID: "0d861409-3dd5-4eed-ada7-e1817b7fb58f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:27:57.528773 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.528752 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:57.528853 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.528807 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret podName:8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:29.528792707 +0000 UTC m=+65.469005979 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret") pod "global-pull-secret-syncer-ht56p" (UID: "8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:27:57.605986 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:57.605952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:57.605986 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:57.605976 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:57.606227 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:57.605967 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:57.606227 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.606077 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:57.606227 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.606176 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:57.606368 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.606254 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:57.629141 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:57.629107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:57.629310 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.629291 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:27:57.629368 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.629316 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:27:57.629368 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.629328 2568 projected.go:196] Error preparing data for projected volume kube-api-access-f87bx for pod openshift-network-diagnostics/network-check-target-gzjf9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:57.629452 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:57.629388 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx podName:8b9dc277-9345-44d1-9f7e-90b5a9f3380e nodeName:}" failed. No retries permitted until 2026-10-31 00:28:29.629372704 +0000 UTC m=+65.569585984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f87bx" (UniqueName: "kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx") pod "network-check-target-gzjf9" (UID: "8b9dc277-9345-44d1-9f7e-90b5a9f3380e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:27:58.068924 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:58.068880 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7x6x2_aad04f16-056a-4115-97a0-e75d9e8b36b7/node-ca/0.log" Oct 31 00:27:59.605962 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.605834 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:27:59.606258 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.605891 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:27:59.606258 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:59.606055 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:27:59.606258 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.605899 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:27:59.606258 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:59.606123 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:27:59.606258 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:27:59.606183 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:27:59.803180 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.803150 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"33149886f66d8562380c0287f5340bced4823de2a488da15fc68906dbe5dcfd6"} Oct 31 00:27:59.803335 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.803216 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"759378fb75a6ae4c78d18b0bc86e999e431c04017b09eabbd3f3f33870d09ed1"} Oct 31 00:27:59.803335 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.803254 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"151f0a992d5e6c09841fe57469ae3ce55c301281e34ea4342513cfc9bbe27e93"} Oct 31 00:27:59.803335 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.803281 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"a5167b7e5671708515fafe278e0ed1faa9bfdde1b348108622446f4c40946789"} Oct 31 00:27:59.803335 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.803312 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"010dfa7a91781efe794ed581a65358c8716e539e76f6f779c79c9486f65d1cda"} Oct 31 00:27:59.803528 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:27:59.803343 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"a6405b5e4a2f9f9cabbb6c46895f8519d54a2dda22dca016fca3331cf04c4830"} Oct 31 00:28:01.606082 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:01.606053 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:01.606579 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:01.606045 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:01.606579 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:01.606167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:01.606579 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:01.606150 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:01.606579 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:01.606261 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:01.606579 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:01.606350 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:02.812298 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:02.812264 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"38a6aeee75f55e116cab6d411ba4e58f89c396d66061be2145197275bb3fa657"} Oct 31 00:28:03.606209 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:03.606179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:03.606386 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:03.606184 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:03.606386 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:03.606283 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:03.606386 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:03.606184 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:03.606386 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:03.606358 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:03.606521 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:03.606424 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:04.819810 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:04.819777 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" event={"ID":"b42b8ff9-0140-4b1c-97f2-131bc141352f","Type":"ContainerStarted","Data":"c460c7924038a4a13f651de5ed7e793b3a45c0bb85e1b931e969b9f710105cf5"} Oct 31 00:28:04.820449 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:04.820064 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:04.820449 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:04.820299 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:04.835880 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:04.835857 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:05.084001 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:05.083890 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" podStartSLOduration=7.635550533 podStartE2EDuration="40.083878409s" totalImagesPullingTime="32.448327876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:05.083402122 +0000 UTC m=+41.023615409" watchObservedRunningTime="2026-10-31 00:28:05.083878409 +0000 UTC m=+41.024091696" Oct 31 00:28:05.606096 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:05.606053 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:05.606096 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:05.606087 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:05.606096 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:05.606095 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:05.606384 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:05.606185 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:05.606384 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:05.606266 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:05.606384 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:05.606338 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:05.821779 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:05.821756 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:05.836914 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:05.836894 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:05.869194 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:05.869122 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m98mn_b75cde1b-aaee-46c8-b956-dcecf3b6afbf/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:06.265755 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:06.265669 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m98mn_b75cde1b-aaee-46c8-b956-dcecf3b6afbf/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:06.467600 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:06.467566 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m98mn_b75cde1b-aaee-46c8-b956-dcecf3b6afbf/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:06.793826 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:06.793782 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m98mn_b75cde1b-aaee-46c8-b956-dcecf3b6afbf/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:07.605955 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:07.605915 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:07.606328 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:07.605988 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:07.606328 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:07.606014 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:07.606328 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:07.606105 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:07.606328 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:07.606135 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:07.606328 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:07.606188 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:09.606243 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:09.606210 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:09.606243 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:09.606230 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:09.606677 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:09.606219 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:09.606677 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:09.606301 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:09.606677 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:09.606383 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:09.606677 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:09.606438 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:11.606206 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:11.606176 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:11.606206 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:11.606197 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:11.606711 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:11.606255 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:11.606711 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:11.606315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:11.606711 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:11.606435 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:11.606711 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:11.606537 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:13.606619 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:13.606576 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:13.607156 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:13.606576 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:13.607156 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:13.606686 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:13.607156 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:13.606584 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:13.607156 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:13.606847 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:13.607156 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:13.606915 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:15.605597 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:15.605563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:15.606079 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:15.605631 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:15.606079 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:15.605656 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:15.606079 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:15.605759 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:15.606079 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:15.605807 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:15.606079 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:15.605888 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:16.595243 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:16.595210 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ht56p"] Oct 31 00:28:16.595392 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:16.595337 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:16.595473 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:16.595451 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:16.598356 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:16.598330 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ccs74"] Oct 31 00:28:16.598486 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:16.598429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:16.598568 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:16.598544 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:16.599262 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:16.599236 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gzjf9"] Oct 31 00:28:16.599390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:16.599327 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:16.599454 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:16.599414 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:18.606203 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:18.606167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:18.606604 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:18.606202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:18.606604 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:18.606167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:18.606604 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:18.606308 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:18.606604 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:18.606393 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:18.606604 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:18.606493 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:20.605737 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.605705 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:20.605737 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.605738 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:20.606196 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.605708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:20.606196 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:20.605812 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ccs74" podUID="0d861409-3dd5-4eed-ada7-e1817b7fb58f" Oct 31 00:28:20.606196 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:20.605891 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gzjf9" podUID="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" Oct 31 00:28:20.606196 ip-10-0-129-8 kubenswrapper[2568]: E1031 00:28:20.605966 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ht56p" podUID="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" Oct 31 00:28:20.777320 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.777262 2568 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeReady" Oct 31 00:28:20.777430 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.777363 2568 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:28:20.924674 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.924643 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5jqlp"] Oct 31 00:28:20.927029 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.927014 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k4z7d"] Oct 31 00:28:20.927147 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.927130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:20.929760 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.929742 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:20.934334 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.934317 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-47ws4"] Oct 31 00:28:20.936996 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.936983 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:20.952660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.952639 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5jqlp"] Oct 31 00:28:20.980594 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.980532 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52f7q\"" Oct 31 00:28:20.980744 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.980571 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:28:20.981022 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.980577 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:28:20.981092 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.980591 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hg2s2\"" Oct 31 00:28:20.981092 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.981080 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k4z7d"] Oct 31 00:28:20.981216 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.981200 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:28:20.984654 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.981344 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:28:20.984654 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.982634 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:28:20.985390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.985370 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:28:20.985471 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.985390 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:28:20.997955 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.997934 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:28:20.998069 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.998017 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:28:20.998350 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:20.998333 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9qgw5\"" Oct 31 00:28:21.008564 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008539 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9d50249-2496-4219-827f-72c9043805fe-crio-socket\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.008664 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008565 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnmsw\" (UniqueName: \"kubernetes.io/projected/a9d50249-2496-4219-827f-72c9043805fe-kube-api-access-vnmsw\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.008664 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395bf6-131d-4558-8bfe-83d994039370-tmp-dir\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.008664 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95dhq\" (UniqueName: \"kubernetes.io/projected/16395bf6-131d-4558-8bfe-83d994039370-kube-api-access-95dhq\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.008836 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9d50249-2496-4219-827f-72c9043805fe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.008887 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008837 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395bf6-131d-4558-8bfe-83d994039370-config-volume\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.008924 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008894 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395bf6-131d-4558-8bfe-83d994039370-metrics-tls\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.008961 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008944 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6mf\" (UniqueName: \"kubernetes.io/projected/204028bd-7b66-4354-bd39-54aaca835224-kube-api-access-dz6mf\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:21.008994 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9d50249-2496-4219-827f-72c9043805fe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.008994 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.008981 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/204028bd-7b66-4354-bd39-54aaca835224-cert\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:21.009090 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.009002 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9d50249-2496-4219-827f-72c9043805fe-data-volume\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.027011 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.026987 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-47ws4"] Oct 31 00:28:21.109338 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395bf6-131d-4558-8bfe-83d994039370-config-volume\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.109466 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395bf6-131d-4558-8bfe-83d994039370-metrics-tls\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.109466 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6mf\" (UniqueName: \"kubernetes.io/projected/204028bd-7b66-4354-bd39-54aaca835224-kube-api-access-dz6mf\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:21.109466 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9d50249-2496-4219-827f-72c9043805fe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.109466 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109394 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/204028bd-7b66-4354-bd39-54aaca835224-cert\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:21.109466 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9d50249-2496-4219-827f-72c9043805fe-data-volume\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.109729 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109535 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9d50249-2496-4219-827f-72c9043805fe-crio-socket\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.109729 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmsw\" (UniqueName: \"kubernetes.io/projected/a9d50249-2496-4219-827f-72c9043805fe-kube-api-access-vnmsw\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.109729 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109659 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395bf6-131d-4558-8bfe-83d994039370-tmp-dir\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.109930 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109715 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95dhq\" (UniqueName: \"kubernetes.io/projected/16395bf6-131d-4558-8bfe-83d994039370-kube-api-access-95dhq\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.110000 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109953 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9d50249-2496-4219-827f-72c9043805fe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.110000 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109959 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9d50249-2496-4219-827f-72c9043805fe-crio-socket\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.110229 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.109824 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9d50249-2496-4219-827f-72c9043805fe-data-volume\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.110313 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.110233 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395bf6-131d-4558-8bfe-83d994039370-tmp-dir\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.110375 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.110313 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395bf6-131d-4558-8bfe-83d994039370-config-volume\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.110496 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.110465 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9d50249-2496-4219-827f-72c9043805fe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.113418 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.113399 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395bf6-131d-4558-8bfe-83d994039370-metrics-tls\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.113491 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.113448 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9d50249-2496-4219-827f-72c9043805fe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.113534 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.113491 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/204028bd-7b66-4354-bd39-54aaca835224-cert\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:21.166676 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.166648 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnmsw\" (UniqueName: \"kubernetes.io/projected/a9d50249-2496-4219-827f-72c9043805fe-kube-api-access-vnmsw\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.166798 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.166779 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95dhq\" (UniqueName: \"kubernetes.io/projected/16395bf6-131d-4558-8bfe-83d994039370-kube-api-access-95dhq\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.166834 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.166782 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6mf\" (UniqueName: \"kubernetes.io/projected/204028bd-7b66-4354-bd39-54aaca835224-kube-api-access-dz6mf\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:21.241599 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.241570 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:21.244639 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.244614 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:21.254555 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.254527 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:21.681456 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.681428 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k4z7d"] Oct 31 00:28:21.686837 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.686814 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5jqlp"] Oct 31 00:28:21.687448 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.687427 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-47ws4"] Oct 31 00:28:21.688730 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:21.688683 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204028bd_7b66_4354_bd39_54aaca835224.slice/crio-2633f481618d7b185b582b1473ccf23b696e7ea663d8656b96ebed351c37134b WatchSource:0}: Error finding container 2633f481618d7b185b582b1473ccf23b696e7ea663d8656b96ebed351c37134b: Status 404 returned error can't find the container with id 2633f481618d7b185b582b1473ccf23b696e7ea663d8656b96ebed351c37134b Oct 31 00:28:21.695269 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:21.695246 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16395bf6_131d_4558_8bfe_83d994039370.slice/crio-1e1473fbc5a0aa9a10f7af5ce005b8c5ba302e4550e8bc369a056f768da5646e WatchSource:0}: Error finding container 1e1473fbc5a0aa9a10f7af5ce005b8c5ba302e4550e8bc369a056f768da5646e: Status 404 returned error can't find the container with id 1e1473fbc5a0aa9a10f7af5ce005b8c5ba302e4550e8bc369a056f768da5646e Oct 31 00:28:21.695841 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:21.695817 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d50249_2496_4219_827f_72c9043805fe.slice/crio-71c7e6739da357f0d3b4841383108c16d6dd1e3a85ea29a4ed29e866439d1053 WatchSource:0}: Error finding container 71c7e6739da357f0d3b4841383108c16d6dd1e3a85ea29a4ed29e866439d1053: Status 404 returned error can't find the container with id 71c7e6739da357f0d3b4841383108c16d6dd1e3a85ea29a4ed29e866439d1053 Oct 31 00:28:21.855336 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.855300 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-47ws4" event={"ID":"a9d50249-2496-4219-827f-72c9043805fe","Type":"ContainerStarted","Data":"0bc10f1714efa54d3668736aaa46b467c1a422640204f1053d99ca98aff9a038"} Oct 31 00:28:21.855503 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.855360 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-47ws4" event={"ID":"a9d50249-2496-4219-827f-72c9043805fe","Type":"ContainerStarted","Data":"71c7e6739da357f0d3b4841383108c16d6dd1e3a85ea29a4ed29e866439d1053"} Oct 31 00:28:21.856464 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.856441 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k4z7d" event={"ID":"204028bd-7b66-4354-bd39-54aaca835224","Type":"ContainerStarted","Data":"2633f481618d7b185b582b1473ccf23b696e7ea663d8656b96ebed351c37134b"} Oct 31 00:28:21.857502 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:21.857484 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jqlp" event={"ID":"16395bf6-131d-4558-8bfe-83d994039370","Type":"ContainerStarted","Data":"1e1473fbc5a0aa9a10f7af5ce005b8c5ba302e4550e8bc369a056f768da5646e"} Oct 31 00:28:22.605970 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.605930 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:22.606129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.606067 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:22.606129 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.606083 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:22.655836 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.655806 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:28:22.655836 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.655828 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:28:22.702321 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.702290 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nwzth\"" Oct 31 00:28:22.702830 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.702327 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:28:22.702830 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.702371 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dwmlr\"" Oct 31 00:28:22.702830 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.702397 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:28:22.702830 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:22.702493 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m2wdr\"" Oct 31 00:28:23.865806 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:23.865762 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-47ws4" event={"ID":"a9d50249-2496-4219-827f-72c9043805fe","Type":"ContainerStarted","Data":"56e5e20d4d2b653e7b90aae183eeed0fbba6ee45096d722fdec3d4adfbcbee33"} Oct 31 00:28:24.871200 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:24.870904 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jqlp" event={"ID":"16395bf6-131d-4558-8bfe-83d994039370","Type":"ContainerStarted","Data":"2f47634bd99791c9a41b86e12ae8bae7b9072dca5148322fa30f417575eb8335"} Oct 31 00:28:24.871200 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:24.871185 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5jqlp" event={"ID":"16395bf6-131d-4558-8bfe-83d994039370","Type":"ContainerStarted","Data":"c28927462945804f95004bb222d55a0e7cc7cae1f89d4b36c31fade03a480c36"} Oct 31 00:28:24.871956 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:24.871220 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:25.014015 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:25.013960 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5jqlp" podStartSLOduration=2.6900398340000002 podStartE2EDuration="5.013937117s" totalImagesPullingTime="2.323897283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:24.982634725 +0000 UTC m=+60.922848012" watchObservedRunningTime="2026-10-31 00:28:25.013937117 +0000 UTC m=+60.954150405" Oct 31 00:28:25.876043 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:25.875995 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-47ws4" event={"ID":"a9d50249-2496-4219-827f-72c9043805fe","Type":"ContainerStarted","Data":"3ef6eb08d2c609163e6c9b3b1d4c87fae3a382fc95dda3a013fc66812c6a6ed6"} Oct 31 00:28:26.034174 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.034123 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-47ws4" podStartSLOduration=2.345338108 podStartE2EDuration="6.034108867s" totalImagesPullingTime="3.688770759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:25.988373973 +0000 UTC m=+61.928587280" watchObservedRunningTime="2026-10-31 00:28:26.034108867 +0000 UTC m=+61.974322155" Oct 31 00:28:26.735401 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.735364 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv"] Oct 31 00:28:26.738759 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.738741 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.788156 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.788105 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-75n8c\"" Oct 31 00:28:26.788156 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.788107 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:28:26.788156 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.788159 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:28:26.788402 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.788243 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:28:26.788402 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.788393 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:28:26.788463 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.788451 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:28:26.798077 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.798054 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv"] Oct 31 00:28:26.836344 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.836320 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5"] Oct 31 00:28:26.839328 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.839314 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:26.844015 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.843986 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mqb6s"] Oct 31 00:28:26.847179 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.847163 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.855402 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.855386 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.855534 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.855409 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.855534 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.855433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.855601 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.855545 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcd45\" (UniqueName: \"kubernetes.io/projected/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-kube-api-access-tcd45\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.879783 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.879756 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k4z7d" event={"ID":"204028bd-7b66-4354-bd39-54aaca835224","Type":"ContainerStarted","Data":"87d0bbd6f8156b7d45bbb4732ce639ef6a211b2b02374e694f711438c2587910"} Oct 31 00:28:26.893624 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.893608 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:28:26.910147 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.910128 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5"] Oct 31 00:28:26.956652 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.956625 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-accelerators-collector-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.956769 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.956669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcd45\" (UniqueName: \"kubernetes.io/projected/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-kube-api-access-tcd45\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.956769 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.956686 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2mc\" (UniqueName: \"kubernetes.io/projected/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-api-access-4p2mc\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:26.956884 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.956866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-tls\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.956934 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.956905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-root\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.956984 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.956948 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:26.957049 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957026 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.957084 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:26.957141 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957101 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-metrics-client-ca\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.957190 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-sys\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.957239 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957195 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-wtmp\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.957239 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957221 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.957333 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957300 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:26.957384 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.957424 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957388 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg94p\" (UniqueName: \"kubernetes.io/projected/d142f31b-3461-4f5a-93e1-088ce6511100-kube-api-access-hg94p\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.957424 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957408 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:26.957541 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957520 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.957658 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957551 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-textfile\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:26.957658 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.957581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:26.958103 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.958087 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.961150 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.961126 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.961225 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.961128 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:26.987901 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.987843 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:28:26.987901 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.987848 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:28:26.987901 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.987887 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:28:26.988083 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.987999 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7hhvp\"" Oct 31 00:28:26.988083 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.988003 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5h47b\"" Oct 31 00:28:26.988083 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.988048 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:28:26.988368 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:26.988355 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:28:27.014025 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.014002 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcd45\" (UniqueName: \"kubernetes.io/projected/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-kube-api-access-tcd45\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:27.053426 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.053410 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:27.058525 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-sys\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058535 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-wtmp\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058589 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058584 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.058763 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058632 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg94p\" (UniqueName: \"kubernetes.io/projected/d142f31b-3461-4f5a-93e1-088ce6511100-kube-api-access-hg94p\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058763 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058627 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-sys\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058763 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.058763 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-textfile\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058763 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058708 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-wtmp\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058997 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.058997 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058873 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-accelerators-collector-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058997 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058905 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2mc\" (UniqueName: \"kubernetes.io/projected/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-api-access-4p2mc\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.058997 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-tls\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.058997 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.058976 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-root\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.059220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.059000 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.059220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.059041 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.059220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.059069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-metrics-client-ca\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.059220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.059072 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.059220 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.059100 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-textfile\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.059435 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.059408 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.059497 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.059378 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-root\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.059881 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.059856 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-accelerators-collector-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.060133 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.060111 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-metrics-client-ca\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.060483 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.060458 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.061542 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.061524 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.061646 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.061585 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.061987 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.061968 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.062510 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.062494 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-tls\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.116666 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.116630 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg94p\" (UniqueName: \"kubernetes.io/projected/d142f31b-3461-4f5a-93e1-088ce6511100-kube-api-access-hg94p\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.117352 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.117306 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2mc\" (UniqueName: \"kubernetes.io/projected/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-api-access-4p2mc\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.155283 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.155259 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:27.161315 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.161299 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:27.170530 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:27.170507 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd142f31b_3461_4f5a_93e1_088ce6511100.slice/crio-42730e787618364ebf636ece433affc2dec265043991ca6c9540f88da6b2fa6a WatchSource:0}: Error finding container 42730e787618364ebf636ece433affc2dec265043991ca6c9540f88da6b2fa6a: Status 404 returned error can't find the container with id 42730e787618364ebf636ece433affc2dec265043991ca6c9540f88da6b2fa6a Oct 31 00:28:27.181152 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.178546 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k4z7d" podStartSLOduration=2.676340242 podStartE2EDuration="7.178524111s" totalImagesPullingTime="4.502183869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:27.176893355 +0000 UTC m=+63.117106643" watchObservedRunningTime="2026-10-31 00:28:27.178524111 +0000 UTC m=+63.118737400" Oct 31 00:28:27.470067 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.470039 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv"] Oct 31 00:28:27.476775 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:27.476751 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c4027c_b9b6_4a0d_a6d0_66f5a76004bf.slice/crio-1ce35bba90356bc4bfffbc2963fe77982ba0636d6e5fa8e8d0db05461ae0bded WatchSource:0}: Error finding container 1ce35bba90356bc4bfffbc2963fe77982ba0636d6e5fa8e8d0db05461ae0bded: Status 404 returned error can't find the container with id 1ce35bba90356bc4bfffbc2963fe77982ba0636d6e5fa8e8d0db05461ae0bded Oct 31 00:28:27.523767 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.523745 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5"] Oct 31 00:28:27.531633 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:27.531609 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75963a1c_a5af_4c12_8adc_f6c5b9aa2e39.slice/crio-95cc97cc04fa989cc8f45aa3e76a34fd6778bb5107d5465644662c66f6c88c80 WatchSource:0}: Error finding container 95cc97cc04fa989cc8f45aa3e76a34fd6778bb5107d5465644662c66f6c88c80: Status 404 returned error can't find the container with id 95cc97cc04fa989cc8f45aa3e76a34fd6778bb5107d5465644662c66f6c88c80 Oct 31 00:28:27.805106 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.805032 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-697bb4b4df-4gpsf"] Oct 31 00:28:27.808064 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.808046 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.859229 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.859193 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:28:27.859388 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.859358 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:28:27.859388 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.859366 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wqtpp\"" Oct 31 00:28:27.859863 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.859841 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:28:27.859863 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.859844 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:28:27.859863 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.859859 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:28:27.860082 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.859886 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:28:27.860082 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.859920 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:28:27.866009 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.865974 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.866340 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.866013 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.866340 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.866113 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.866340 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.866142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.866340 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.866208 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.866340 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.866240 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.866340 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.866333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.866577 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.866363 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.883793 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.883766 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mqb6s" event={"ID":"d142f31b-3461-4f5a-93e1-088ce6511100","Type":"ContainerStarted","Data":"42730e787618364ebf636ece433affc2dec265043991ca6c9540f88da6b2fa6a"} Oct 31 00:28:27.885627 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.885604 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" event={"ID":"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf","Type":"ContainerStarted","Data":"fe29543dce6c50713a91bbd44be45b7b14364d4784731d16c13d0b85d990decd"} Oct 31 00:28:27.885732 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.885657 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" event={"ID":"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf","Type":"ContainerStarted","Data":"5322c82cff9e8fa0f9b79b9a875affe001e4525835807cb5572fd0ceea5fcc23"} Oct 31 00:28:27.885732 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.885687 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" event={"ID":"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf","Type":"ContainerStarted","Data":"1ce35bba90356bc4bfffbc2963fe77982ba0636d6e5fa8e8d0db05461ae0bded"} Oct 31 00:28:27.886795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.886773 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" event={"ID":"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39","Type":"ContainerStarted","Data":"95cc97cc04fa989cc8f45aa3e76a34fd6778bb5107d5465644662c66f6c88c80"} Oct 31 00:28:27.906846 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.906816 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:28:27.907674 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.907656 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-697bb4b4df-4gpsf"] Oct 31 00:28:27.967312 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.967280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.967468 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.967340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.967468 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.967368 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.967468 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.967412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.967468 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.967439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.967724 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.967476 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.967724 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.967508 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.967724 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.967547 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.968061 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.968033 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.968186 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.968128 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.968186 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.968172 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.968742 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.968723 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.968793 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.968764 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.970547 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.970524 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:27.970612 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:27.970601 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:28.025526 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:28.025498 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:28.119099 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:28.119065 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:28.497388 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:28.497320 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-697bb4b4df-4gpsf"] Oct 31 00:28:28.503172 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:28.503150 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1368083a_8041_45bb_8e12_93f10e05eb15.slice/crio-a5854abe3749dd55c8aeb07c3d7894ed8a35e88d2fa28f56418efb86b5b309fd WatchSource:0}: Error finding container a5854abe3749dd55c8aeb07c3d7894ed8a35e88d2fa28f56418efb86b5b309fd: Status 404 returned error can't find the container with id a5854abe3749dd55c8aeb07c3d7894ed8a35e88d2fa28f56418efb86b5b309fd Oct 31 00:28:28.891807 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:28.891769 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697bb4b4df-4gpsf" event={"ID":"1368083a-8041-45bb-8e12-93f10e05eb15","Type":"ContainerStarted","Data":"a5854abe3749dd55c8aeb07c3d7894ed8a35e88d2fa28f56418efb86b5b309fd"} Oct 31 00:28:29.055279 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.055247 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:28:29.059127 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.059103 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.125317 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.125292 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:28:29.161186 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.161117 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:28:29.161325 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.161186 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:28:29.161325 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.161249 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:28:29.161325 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.161262 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:28:29.161325 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.161317 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:28:29.161498 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.161377 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:28:29.161498 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.161432 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tjsbm\"" Oct 31 00:28:29.170011 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.169963 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:28:29.170011 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.169961 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:28:29.170192 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.170074 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:28:29.178375 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zs9\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178483 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178394 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178549 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178487 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178599 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178552 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178645 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178625 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178729 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178710 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178809 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178867 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178904 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178935 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.178966 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178947 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.179010 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.179010 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.178999 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280310 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280428 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280428 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280362 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280428 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280388 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280428 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280418 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280452 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zs9\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280568 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280593 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280947 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.280947 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.280713 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.281243 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.281219 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.282758 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.282500 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.283002 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.282959 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.283942 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.283913 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.289559 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.289538 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.290748 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.290727 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.293365 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.293200 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.293365 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.293282 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.293365 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.293290 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.293365 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.293322 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.293365 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.293327 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.293567 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.293442 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.343507 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.343468 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zs9\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.388888 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.388859 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:29.471322 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.471292 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4"] Oct 31 00:28:29.476526 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.476493 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.528644 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.528539 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:28:29.528644 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.528575 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:28:29.528644 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.528594 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:28:29.528644 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.528539 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:28:29.528644 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.528610 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qnk5n\"" Oct 31 00:28:29.528644 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.528551 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dqugrurpbd40l\"" Oct 31 00:28:29.528644 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.528651 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:28:29.537682 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.537656 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4"] Oct 31 00:28:29.583062 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0910c96-18d0-49b2-861d-0ec2a5793047-tmp\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.583062 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583055 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.583189 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.583189 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-grpc-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.583280 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:29.583371 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583309 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.583371 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:29.583469 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5k99\" (UniqueName: \"kubernetes.io/projected/d0910c96-18d0-49b2-861d-0ec2a5793047-kube-api-access-q5k99\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.583469 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.583560 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583540 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.583754 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.583732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0910c96-18d0-49b2-861d-0ec2a5793047-metrics-client-ca\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.631730 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.631705 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:28:29.631844 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.631746 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:28:29.636171 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.636150 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:29.636292 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.636194 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:29.684674 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.684866 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684682 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.684866 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0910c96-18d0-49b2-861d-0ec2a5793047-metrics-client-ca\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.684866 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0910c96-18d0-49b2-861d-0ec2a5793047-tmp\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.684866 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.684866 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.684866 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-grpc-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.685203 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:29.685203 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684913 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.685203 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.684951 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5k99\" (UniqueName: \"kubernetes.io/projected/d0910c96-18d0-49b2-861d-0ec2a5793047-kube-api-access-q5k99\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.685288 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.685233 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0910c96-18d0-49b2-861d-0ec2a5793047-tmp\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.685707 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.685667 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0910c96-18d0-49b2-861d-0ec2a5793047-metrics-client-ca\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.687833 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.687795 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.688210 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.688186 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.688382 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.688361 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-grpc-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.688834 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.688815 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.688984 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.688968 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.689562 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.689542 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.736229 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.736161 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:28:29.752392 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.752370 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5k99\" (UniqueName: \"kubernetes.io/projected/d0910c96-18d0-49b2-861d-0ec2a5793047-kube-api-access-q5k99\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.793296 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.793273 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:28:29.798214 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.798196 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:29.813186 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.813167 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:29.841734 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.841642 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:28:29.847619 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:29.847594 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c01729b_d9cb_44ff_bcfd_3db050a479b8.slice/crio-adf94de2be2239c94b6926e78a9413109c3c9e0e7ab2706bfa14317941079985 WatchSource:0}: Error finding container adf94de2be2239c94b6926e78a9413109c3c9e0e7ab2706bfa14317941079985: Status 404 returned error can't find the container with id adf94de2be2239c94b6926e78a9413109c3c9e0e7ab2706bfa14317941079985 Oct 31 00:28:29.869336 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.869241 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m2wdr\"" Oct 31 00:28:29.872130 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.872057 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:29.872232 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.872159 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nwzth\"" Oct 31 00:28:29.874396 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.874380 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:29.877385 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.877372 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dwmlr\"" Oct 31 00:28:29.880485 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.880363 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:29.898990 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.898956 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" event={"ID":"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf","Type":"ContainerStarted","Data":"650d1d21375a0e2e18e872fe4b4a646a0e2dbab6d4369a590ac001e46f429387"} Oct 31 00:28:29.901062 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.901035 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerStarted","Data":"adf94de2be2239c94b6926e78a9413109c3c9e0e7ab2706bfa14317941079985"} Oct 31 00:28:29.904250 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.903539 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" event={"ID":"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39","Type":"ContainerStarted","Data":"4a5dfdc7cc0de18e5b613e6ffa46c4d481bad8c76ad5599c4a05337dc02ed306"} Oct 31 00:28:29.904250 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.903593 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" event={"ID":"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39","Type":"ContainerStarted","Data":"5a9540b68176fbbb48e52c1efeb64d6b4f62f43d1430e23dd8a4de99261e5fe7"} Oct 31 00:28:29.904250 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.903624 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" event={"ID":"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39","Type":"ContainerStarted","Data":"07e1fb2fd88c9ab00c286d9a810c8df872526b88c24d8e8b7033ce0612be1559"} Oct 31 00:28:29.905916 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.905884 2568 generic.go:356] "Generic (PLEG): container finished" podID="d142f31b-3461-4f5a-93e1-088ce6511100" containerID="f9fae05c0daf26caca56cc9e2078979df05366472eb9053101ee2d12718467cd" exitCode=0 Oct 31 00:28:29.905977 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:29.905923 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mqb6s" event={"ID":"d142f31b-3461-4f5a-93e1-088ce6511100","Type":"ContainerDied","Data":"f9fae05c0daf26caca56cc9e2078979df05366472eb9053101ee2d12718467cd"} Oct 31 00:28:30.032598 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.032546 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" podStartSLOduration=2.060482435 podStartE2EDuration="4.032529953s" totalImagesPullingTime="1.972047518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:30.031472513 +0000 UTC m=+65.971685801" watchObservedRunningTime="2026-10-31 00:28:30.032529953 +0000 UTC m=+65.972743242" Oct 31 00:28:30.200829 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.200767 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" podStartSLOduration=2.271397528 podStartE2EDuration="4.20074752s" totalImagesPullingTime="1.929349992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:30.155176793 +0000 UTC m=+66.095390082" watchObservedRunningTime="2026-10-31 00:28:30.20074752 +0000 UTC m=+66.140960809" Oct 31 00:28:30.201120 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.201103 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4"] Oct 31 00:28:30.207971 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:30.207918 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0910c96_18d0_49b2_861d_0ec2a5793047.slice/crio-91142f3525707602388be1bccce15518cceb9860c4cf2c39f4243cdea857d465 WatchSource:0}: Error finding container 91142f3525707602388be1bccce15518cceb9860c4cf2c39f4243cdea857d465: Status 404 returned error can't find the container with id 91142f3525707602388be1bccce15518cceb9860c4cf2c39f4243cdea857d465 Oct 31 00:28:30.311615 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.311532 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ccs74"] Oct 31 00:28:30.313637 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.313595 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ht56p"] Oct 31 00:28:30.319459 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:30.319432 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c091a1b_9f15_4dcf_a55a_4d2c07ddcdc5.slice/crio-daa5e7ad4eee821437d8f097c3b127c37c84d43fc2333f1f50dbc7a8e6fb6830 WatchSource:0}: Error finding container daa5e7ad4eee821437d8f097c3b127c37c84d43fc2333f1f50dbc7a8e6fb6830: Status 404 returned error can't find the container with id daa5e7ad4eee821437d8f097c3b127c37c84d43fc2333f1f50dbc7a8e6fb6830 Oct 31 00:28:30.320817 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:30.320778 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d861409_3dd5_4eed_ada7_e1817b7fb58f.slice/crio-16f55a4d9dc60a00d646c17530e5b9a9468c4e2cc45988cace33d484ca02c6fb WatchSource:0}: Error finding container 16f55a4d9dc60a00d646c17530e5b9a9468c4e2cc45988cace33d484ca02c6fb: Status 404 returned error can't find the container with id 16f55a4d9dc60a00d646c17530e5b9a9468c4e2cc45988cace33d484ca02c6fb Oct 31 00:28:30.339925 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.339903 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gzjf9"] Oct 31 00:28:30.352619 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:30.352592 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9dc277_9345_44d1_9f7e_90b5a9f3380e.slice/crio-c207d12fda7f63e8b83e7ad118e1da170cd71ff8b630ad64f7616d0502cd827f WatchSource:0}: Error finding container c207d12fda7f63e8b83e7ad118e1da170cd71ff8b630ad64f7616d0502cd827f: Status 404 returned error can't find the container with id c207d12fda7f63e8b83e7ad118e1da170cd71ff8b630ad64f7616d0502cd827f Oct 31 00:28:30.912366 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.912294 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ht56p" event={"ID":"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5","Type":"ContainerStarted","Data":"daa5e7ad4eee821437d8f097c3b127c37c84d43fc2333f1f50dbc7a8e6fb6830"} Oct 31 00:28:30.913859 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.913827 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ccs74" event={"ID":"0d861409-3dd5-4eed-ada7-e1817b7fb58f","Type":"ContainerStarted","Data":"16f55a4d9dc60a00d646c17530e5b9a9468c4e2cc45988cace33d484ca02c6fb"} Oct 31 00:28:30.915105 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.915084 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gzjf9" event={"ID":"8b9dc277-9345-44d1-9f7e-90b5a9f3380e","Type":"ContainerStarted","Data":"c207d12fda7f63e8b83e7ad118e1da170cd71ff8b630ad64f7616d0502cd827f"} Oct 31 00:28:30.917390 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.917369 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mqb6s" event={"ID":"d142f31b-3461-4f5a-93e1-088ce6511100","Type":"ContainerStarted","Data":"fd2765ce19b454484ad7ecf1210a8692f70ad494bd076da56907d5a91a36397c"} Oct 31 00:28:30.917497 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.917411 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mqb6s" event={"ID":"d142f31b-3461-4f5a-93e1-088ce6511100","Type":"ContainerStarted","Data":"82131e38ae6664e142294bdaa7887ddab1d27443bbc7238ec2ab75eab3e77b24"} Oct 31 00:28:30.918715 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:30.918676 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" event={"ID":"d0910c96-18d0-49b2-861d-0ec2a5793047","Type":"ContainerStarted","Data":"91142f3525707602388be1bccce15518cceb9860c4cf2c39f4243cdea857d465"} Oct 31 00:28:31.034576 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.034520 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mqb6s" podStartSLOduration=2.906733077 podStartE2EDuration="5.034504357s" totalImagesPullingTime="2.047835902s" totalInitContainerRuntime="79.935378ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:31.03219533 +0000 UTC m=+66.972408618" watchObservedRunningTime="2026-10-31 00:28:31.034504357 +0000 UTC m=+66.974717665" Oct 31 00:28:31.574234 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.574196 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh"] Oct 31 00:28:31.587504 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.587474 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:31.647346 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.647314 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x4kkw\"" Oct 31 00:28:31.647346 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.647314 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:28:31.659958 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.659925 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh"] Oct 31 00:28:31.703817 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.703780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdd6j\" (UniqueName: \"kubernetes.io/projected/fda3848b-9d0b-4832-8789-862920d484b8-kube-api-access-vdd6j\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:31.704002 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.703883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fda3848b-9d0b-4832-8789-862920d484b8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:31.796403 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.796364 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d46547967-pkq4f"] Oct 31 00:28:31.805035 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.805002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd6j\" (UniqueName: \"kubernetes.io/projected/fda3848b-9d0b-4832-8789-862920d484b8-kube-api-access-vdd6j\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:31.805177 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.805068 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fda3848b-9d0b-4832-8789-862920d484b8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:31.815544 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.815511 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fda3848b-9d0b-4832-8789-862920d484b8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:31.816711 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.816672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:31.859070 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.859037 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d46547967-pkq4f"] Oct 31 00:28:31.868533 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.868459 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd6j\" (UniqueName: \"kubernetes.io/projected/fda3848b-9d0b-4832-8789-862920d484b8-kube-api-access-vdd6j\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:31.868866 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.868737 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:28:31.868866 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.868828 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:28:31.869765 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.869743 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:28:31.869961 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.869945 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:28:31.870025 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.870007 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kqch6\"" Oct 31 00:28:31.870025 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.870012 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3ank9btekulg4\"" Oct 31 00:28:31.906080 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.906045 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f65619e-5f82-4313-8a35-572fbece3755-audit-log\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:31.906230 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.906092 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-client-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:31.906230 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.906137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-tls\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:31.906230 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.906183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:31.906230 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.906207 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-client-certs\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:31.906230 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.906226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-metrics-server-audit-profiles\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:31.906492 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.906251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7htb\" (UniqueName: \"kubernetes.io/projected/5f65619e-5f82-4313-8a35-572fbece3755-kube-api-access-h7htb\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:31.913415 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:31.912617 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:32.007570 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.007543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f65619e-5f82-4313-8a35-572fbece3755-audit-log\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.007726 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.007574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-client-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.007726 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.007649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-tls\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.007850 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.007737 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.007850 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.007790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-client-certs\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.007850 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.007826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-metrics-server-audit-profiles\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.007991 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.007854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7htb\" (UniqueName: \"kubernetes.io/projected/5f65619e-5f82-4313-8a35-572fbece3755-kube-api-access-h7htb\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.008875 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.008850 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f65619e-5f82-4313-8a35-572fbece3755-audit-log\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.011047 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.010810 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.019514 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.015565 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-metrics-server-audit-profiles\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.019514 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.016579 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-client-certs\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.019514 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.018296 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-tls\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.023392 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.023323 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-client-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.077538 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.077485 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7htb\" (UniqueName: \"kubernetes.io/projected/5f65619e-5f82-4313-8a35-572fbece3755-kube-api-access-h7htb\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.149593 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.149320 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:32.521493 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.521412 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh"] Oct 31 00:28:32.533060 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:32.533000 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda3848b_9d0b_4832_8789_862920d484b8.slice/crio-2b9f62e5b02c9a2156d4509505e6023213fc1cb28842032fef23adfcdbe50853 WatchSource:0}: Error finding container 2b9f62e5b02c9a2156d4509505e6023213fc1cb28842032fef23adfcdbe50853: Status 404 returned error can't find the container with id 2b9f62e5b02c9a2156d4509505e6023213fc1cb28842032fef23adfcdbe50853 Oct 31 00:28:32.573277 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.573250 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d46547967-pkq4f"] Oct 31 00:28:32.581453 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:32.581429 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f65619e_5f82_4313_8a35_572fbece3755.slice/crio-67e9aa3626c4c2b2c7908325b4067ccf200a80786073f9ba611fb17a551c986c WatchSource:0}: Error finding container 67e9aa3626c4c2b2c7908325b4067ccf200a80786073f9ba611fb17a551c986c: Status 404 returned error can't find the container with id 67e9aa3626c4c2b2c7908325b4067ccf200a80786073f9ba611fb17a551c986c Oct 31 00:28:32.930578 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.930511 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" event={"ID":"5f65619e-5f82-4313-8a35-572fbece3755","Type":"ContainerStarted","Data":"67e9aa3626c4c2b2c7908325b4067ccf200a80786073f9ba611fb17a551c986c"} Oct 31 00:28:32.933009 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.932952 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" event={"ID":"fda3848b-9d0b-4832-8789-862920d484b8","Type":"ContainerStarted","Data":"2b9f62e5b02c9a2156d4509505e6023213fc1cb28842032fef23adfcdbe50853"} Oct 31 00:28:32.936553 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.936523 2568 generic.go:356] "Generic (PLEG): container finished" podID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerID="122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d" exitCode=0 Oct 31 00:28:32.936658 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:32.936572 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerDied","Data":"122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d"} Oct 31 00:28:33.945988 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:33.945952 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ccs74" event={"ID":"0d861409-3dd5-4eed-ada7-e1817b7fb58f","Type":"ContainerStarted","Data":"9e5af53f7def7fa1a6a7b224363c0fe75f6a53ab23d341d75e7c4352331b9500"} Oct 31 00:28:33.946468 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:33.946019 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ccs74" event={"ID":"0d861409-3dd5-4eed-ada7-e1817b7fb58f","Type":"ContainerStarted","Data":"f13c3d7d71e903b41d6b1a31439cfb639a409ca5b292b4f127f52aed6fa6490d"} Oct 31 00:28:34.058256 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:34.057067 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ccs74" podStartSLOduration=66.254845095 podStartE2EDuration="1m9.057047633s" totalImagesPullingTime="2.802202538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:34.056191995 +0000 UTC m=+69.996405283" watchObservedRunningTime="2026-10-31 00:28:34.057047633 +0000 UTC m=+69.997260922" Oct 31 00:28:34.877841 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:34.877812 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:35.982566 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:35.981803 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697bb4b4df-4gpsf" event={"ID":"1368083a-8041-45bb-8e12-93f10e05eb15","Type":"ContainerStarted","Data":"50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67"} Oct 31 00:28:35.985661 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:35.985133 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" event={"ID":"5f65619e-5f82-4313-8a35-572fbece3755","Type":"ContainerStarted","Data":"f1b9d3285ae17d169b9744eefc8e13aae510e9e040ed56dff87f23834b7d14c2"} Oct 31 00:28:35.991523 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:35.991445 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" event={"ID":"d0910c96-18d0-49b2-861d-0ec2a5793047","Type":"ContainerStarted","Data":"826c6cd535bfeefb7b8a9d4867c536a3de844371c7b4f10b91c438bcfa33389d"} Oct 31 00:28:36.139852 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.139727 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-697bb4b4df-4gpsf" podStartSLOduration=1.936433864 podStartE2EDuration="9.13970688s" totalImagesPullingTime="7.203273016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:36.139112164 +0000 UTC m=+72.079325454" watchObservedRunningTime="2026-10-31 00:28:36.13970688 +0000 UTC m=+72.079920163" Oct 31 00:28:36.258453 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.258340 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" podStartSLOduration=2.123808427 podStartE2EDuration="5.258324273s" totalImagesPullingTime="3.134515846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:36.256930428 +0000 UTC m=+72.197143739" watchObservedRunningTime="2026-10-31 00:28:36.258324273 +0000 UTC m=+72.198537560" Oct 31 00:28:36.269888 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.269851 2568 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:28:36.273564 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.273537 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.324559 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.324519 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:28:36.324907 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.324887 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9q7lj\"" Oct 31 00:28:36.325034 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.324956 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:28:36.325034 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.324966 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:28:36.325034 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.324984 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:28:36.325034 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.325031 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:28:36.327707 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.327658 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:28:36.356230 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4gq2\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356309 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356342 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356406 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356434 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356547 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.356660 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.357051 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.357051 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.357051 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356793 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.357051 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356838 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.357051 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.356889 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.357051 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.357033 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.378659 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.378626 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:28:36.425982 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.425878 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:28:36.458240 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458376 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458376 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458376 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458549 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458380 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458549 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4gq2\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458549 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458727 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458727 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458727 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458662 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458897 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458805 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458897 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458869 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458993 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458900 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458993 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458993 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458958 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.458993 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.458986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459347 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459042 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459347 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459347 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459347 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459347 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459201 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459656 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459345 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459656 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459356 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459656 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459450 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.459889 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.459868 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.460065 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.460027 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.461456 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.461242 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.462799 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.462357 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.462799 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.462424 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.463767 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.463725 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.464265 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.464249 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.473159 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.473139 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:28:36.473260 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.473195 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:28:36.473260 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.473219 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:28:36.473601 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.473571 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-djuj1hhvgpfmh\"" Oct 31 00:28:36.473601 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.473577 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:28:36.473769 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.473604 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:28:36.473769 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.473650 2568 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:28:36.473769 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.473732 2568 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:28:36.483459 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.483068 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.483459 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.483107 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.483610 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.483552 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.487816 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.486352 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.487816 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.486432 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.487816 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.486830 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.487816 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.487563 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.488023 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.487946 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.521080 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.521053 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4gq2\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.599028 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.598803 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:36.999221 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.999183 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerStarted","Data":"885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06"} Oct 31 00:28:36.999640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.999274 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerStarted","Data":"fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293"} Oct 31 00:28:36.999640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.999315 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerStarted","Data":"31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2"} Oct 31 00:28:36.999640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.999356 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerStarted","Data":"87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3"} Oct 31 00:28:36.999640 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:36.999381 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerStarted","Data":"ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971"} Oct 31 00:28:37.001762 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.001734 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" event={"ID":"d0910c96-18d0-49b2-861d-0ec2a5793047","Type":"ContainerStarted","Data":"72ca2f441421e3b8e6b36ed905da07b575cc8a8209108f0c615e351463c9a947"} Oct 31 00:28:37.001890 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.001811 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" event={"ID":"d0910c96-18d0-49b2-861d-0ec2a5793047","Type":"ContainerStarted","Data":"82c813edbdbfc4b2b2ebb7410b8b4498dae57dc2b139b50339c7d993a866363e"} Oct 31 00:28:37.003434 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.003406 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gzjf9" event={"ID":"8b9dc277-9345-44d1-9f7e-90b5a9f3380e","Type":"ContainerStarted","Data":"e2bea54f45fced12e9154d9fdf46a0a5be3d4e75d5c236f39ad35c8ac0967337"} Oct 31 00:28:37.003550 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.003518 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:37.005179 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.005157 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" event={"ID":"fda3848b-9d0b-4832-8789-862920d484b8","Type":"ContainerStarted","Data":"91468c7af3d0fd72105a78b761849f3497bafc30a4aa862bea9fcf8fc9c9eb1e"} Oct 31 00:28:37.005897 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.005877 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:37.011554 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.011536 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:37.051795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.051725 2568 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:28:37.057923 ip-10-0-129-8 kubenswrapper[2568]: W1031 00:28:37.057891 2568 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11796f3e_c26f_4453_81fc_b50c5ebfa01e.slice/crio-d48dfd49ce4b1e5cd6d20bfa6e1d12ee1f821af1f6a7f8b8d21a279656cca0fd WatchSource:0}: Error finding container d48dfd49ce4b1e5cd6d20bfa6e1d12ee1f821af1f6a7f8b8d21a279656cca0fd: Status 404 returned error can't find the container with id d48dfd49ce4b1e5cd6d20bfa6e1d12ee1f821af1f6a7f8b8d21a279656cca0fd Oct 31 00:28:37.115998 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.115954 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gzjf9" podStartSLOduration=65.722261455 podStartE2EDuration="1m12.115939522s" totalImagesPullingTime="6.393678067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:37.114874277 +0000 UTC m=+73.055087563" watchObservedRunningTime="2026-10-31 00:28:37.115939522 +0000 UTC m=+73.056152809" Oct 31 00:28:37.839527 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.839500 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:37.964561 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:37.964499 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" podStartSLOduration=3.587434446 podStartE2EDuration="6.964482661s" totalImagesPullingTime="3.377048215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:37.22479575 +0000 UTC m=+73.165009037" watchObservedRunningTime="2026-10-31 00:28:37.964482661 +0000 UTC m=+73.904695948" Oct 31 00:28:38.013795 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.013763 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerStarted","Data":"5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9"} Oct 31 00:28:38.017567 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.017535 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" event={"ID":"d0910c96-18d0-49b2-861d-0ec2a5793047","Type":"ContainerStarted","Data":"e480332ea6a854abfe1cafd0e2e05fd161eb8ddc3ed8aba952b511c61005fca9"} Oct 31 00:28:38.017747 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.017616 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" event={"ID":"d0910c96-18d0-49b2-861d-0ec2a5793047","Type":"ContainerStarted","Data":"66fc48d858664870403105611b28536e80c485692ca33ae30ac960621d717ba1"} Oct 31 00:28:38.017747 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.017660 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" event={"ID":"d0910c96-18d0-49b2-861d-0ec2a5793047","Type":"ContainerStarted","Data":"45247d999fb52117ebce38e86ff07154ab18263860b487cd54ebb9af8cf9e2d4"} Oct 31 00:28:38.017747 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.017725 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:38.019989 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.019961 2568 generic.go:356] "Generic (PLEG): container finished" podID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerID="6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9" exitCode=0 Oct 31 00:28:38.020130 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.020048 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerDied","Data":"6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9"} Oct 31 00:28:38.020130 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.020103 2568 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerStarted","Data":"d48dfd49ce4b1e5cd6d20bfa6e1d12ee1f821af1f6a7f8b8d21a279656cca0fd"} Oct 31 00:28:38.119349 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.119280 2568 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:38.119492 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.119357 2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:38.124250 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.124230 2568 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:38.267716 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.267627 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.635070716 podStartE2EDuration="10.267608615s" totalImagesPullingTime="6.618940035s" totalInitContainerRuntime="13.597864ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:38.148269014 +0000 UTC m=+74.088482302" watchObservedRunningTime="2026-10-31 00:28:38.267608615 +0000 UTC m=+74.207821907" Oct 31 00:28:38.268557 ip-10-0-129-8 kubenswrapper[2568]: I1031 00:28:38.268522 2568 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" podStartSLOduration=2.164224629 podStartE2EDuration="9.268513273s" totalImagesPullingTime="7.104288644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:38.266390473 +0000 UTC m=+74.206603780" watchObservedRunningTime="2026-10-31 00:28:38.268513273 +0000 UTC m=+74.208726561" Oct 31 00:28:38.407679 ip-10-0-129-8 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:28:38.421175 ip-10-0-129-8 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:28:38.421357 ip-10-0-129-8 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:28:38.421540 ip-10-0-129-8 systemd[1]: kubelet.service: Consumed 4.223s CPU time. Oct 31 00:28:38.432148 ip-10-0-129-8 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:28:38.495786 ip-10-0-129-8 kubenswrapper[5719]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:38.495786 ip-10-0-129-8 kubenswrapper[5719]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:28:38.495786 ip-10-0-129-8 kubenswrapper[5719]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:38.495786 ip-10-0-129-8 kubenswrapper[5719]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:28:38.497820 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497745 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:38.497820 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497819 5719 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:38.497820 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497822 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497825 5719 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497828 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497831 5719 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497833 5719 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497837 5719 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497839 5719 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497842 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497845 5719 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497847 5719 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497850 5719 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497865 5719 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497870 5719 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497874 5719 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497878 5719 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497881 5719 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497884 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497886 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497889 5719 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:38.497912 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497892 5719 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497894 5719 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497897 5719 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497899 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497902 5719 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497904 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497907 5719 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497909 5719 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497912 5719 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497915 5719 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497917 5719 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497920 5719 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497923 5719 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497925 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497928 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497931 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497933 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497936 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497938 5719 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497940 5719 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:38.498389 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497943 5719 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497945 5719 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497948 5719 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497951 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497953 5719 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497963 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497966 5719 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497968 5719 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497971 5719 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497974 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497976 5719 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497979 5719 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497982 5719 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497984 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497987 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497990 5719 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497993 5719 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497995 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.497998 5719 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498000 5719 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:38.498958 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498002 5719 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498005 5719 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498007 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498010 5719 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498012 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498014 5719 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498017 5719 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498019 5719 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498022 5719 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498024 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498027 5719 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498029 5719 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498032 5719 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498035 5719 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498037 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498040 5719 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498042 5719 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498045 5719 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498048 5719 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:38.499455 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498050 5719 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498052 5719 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498055 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498058 5719 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498062 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498066 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498069 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498072 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498074 5719 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498078 5719 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498081 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498083 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498085 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498088 5719 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498090 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498093 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498095 5719 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498098 5719 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498100 5719 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498103 5719 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:38.499935 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498105 5719 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498109 5719 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498111 5719 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498114 5719 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498116 5719 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498119 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.498996 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499007 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499011 5719 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499015 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499019 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499023 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499028 5719 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499033 5719 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499037 5719 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499041 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499046 5719 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499050 5719 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499053 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499057 5719 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:38.500414 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499061 5719 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499066 5719 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499068 5719 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499071 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499074 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499077 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499080 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499082 5719 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499085 5719 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499087 5719 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499090 5719 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499093 5719 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499095 5719 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499098 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499100 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499103 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499106 5719 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499111 5719 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499116 5719 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:38.500940 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499119 5719 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499122 5719 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499126 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499129 5719 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499131 5719 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499134 5719 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499137 5719 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499140 5719 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499143 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499145 5719 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499148 5719 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499150 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499153 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499155 5719 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499158 5719 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499160 5719 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499163 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499166 5719 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499168 5719 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499171 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:38.501401 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499174 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499177 5719 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499179 5719 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499182 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499184 5719 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499187 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499189 5719 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499192 5719 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499194 5719 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499197 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499200 5719 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499202 5719 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499205 5719 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499208 5719 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499210 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499230 5719 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499233 5719 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499237 5719 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499241 5719 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:38.501909 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499244 5719 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499247 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499250 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499252 5719 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499255 5719 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499258 5719 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499260 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499263 5719 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499265 5719 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499268 5719 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499271 5719 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499273 5719 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499276 5719 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499278 5719 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499281 5719 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499283 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499286 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499289 5719 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499291 5719 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:38.502476 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499293 5719 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499296 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499299 5719 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499301 5719 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499304 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499307 5719 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499309 5719 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499312 5719 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499315 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499317 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499320 5719 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499324 5719 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499327 5719 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499329 5719 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499332 5719 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499822 5719 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499828 5719 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499831 5719 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499833 5719 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499836 5719 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:38.502969 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499838 5719 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499841 5719 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499844 5719 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499846 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499848 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499851 5719 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499854 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499856 5719 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499859 5719 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499861 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499868 5719 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499870 5719 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499873 5719 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499876 5719 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499878 5719 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499881 5719 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499883 5719 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499886 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499889 5719 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499892 5719 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:38.503491 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499895 5719 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499897 5719 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499900 5719 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499902 5719 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499905 5719 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499907 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499910 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499913 5719 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499917 5719 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499921 5719 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499925 5719 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499928 5719 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499931 5719 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499934 5719 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499936 5719 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499939 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499941 5719 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499944 5719 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499946 5719 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:38.504029 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499949 5719 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499951 5719 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499954 5719 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499958 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499960 5719 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499963 5719 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499965 5719 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499968 5719 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499970 5719 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499973 5719 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499975 5719 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499978 5719 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499981 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499983 5719 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499986 5719 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499988 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499991 5719 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499993 5719 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499996 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.499998 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:38.504480 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500001 5719 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500012 5719 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500015 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500018 5719 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500021 5719 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500024 5719 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500026 5719 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500029 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500031 5719 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500034 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500036 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500038 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500041 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500043 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500046 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500049 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500052 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500054 5719 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500057 5719 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:38.505081 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500059 5719 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500062 5719 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500064 5719 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500067 5719 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500069 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500072 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500074 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500076 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500079 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500081 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500084 5719 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500086 5719 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500089 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500091 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500094 5719 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500096 5719 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500099 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500102 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500104 5719 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:38.505597 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500107 5719 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500109 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500111 5719 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500115 5719 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500212 5719 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500223 5719 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500233 5719 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500240 5719 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500246 5719 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500251 5719 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500256 5719 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500260 5719 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500263 5719 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500266 5719 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500269 5719 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500273 5719 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500275 5719 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500278 5719 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500281 5719 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500284 5719 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500287 5719 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500290 5719 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500294 5719 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500297 5719 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:28:38.506325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500300 5719 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500303 5719 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500306 5719 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500310 5719 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500314 5719 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500317 5719 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500320 5719 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500323 5719 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500326 5719 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500329 5719 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500332 5719 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500334 5719 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500338 5719 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500341 5719 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500344 5719 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500347 5719 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500350 5719 flags.go:64] FLAG: --enable-server="true" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500354 5719 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500358 5719 flags.go:64] FLAG: --event-burst="100" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500361 5719 flags.go:64] FLAG: --event-qps="50" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500364 5719 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500367 5719 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500370 5719 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500374 5719 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:28:38.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500377 5719 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500380 5719 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500383 5719 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500388 5719 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500391 5719 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500394 5719 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500396 5719 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500399 5719 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500402 5719 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500405 5719 flags.go:64] FLAG: --feature-gates="" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500409 5719 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500412 5719 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500415 5719 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500418 5719 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500422 5719 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500425 5719 flags.go:64] FLAG: --help="false" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500428 5719 flags.go:64] FLAG: --hostname-override="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500431 5719 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500434 5719 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500437 5719 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500440 5719 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500444 5719 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500447 5719 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:28:38.507790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500450 5719 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500452 5719 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500456 5719 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500459 5719 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500462 5719 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500465 5719 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500468 5719 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500474 5719 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500477 5719 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500480 5719 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500483 5719 flags.go:64] FLAG: --lock-file="" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500486 5719 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500493 5719 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500496 5719 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500501 5719 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500504 5719 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500507 5719 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500510 5719 flags.go:64] FLAG: --logging-format="text" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500513 5719 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500516 5719 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500519 5719 flags.go:64] FLAG: --manifest-url="" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500522 5719 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500526 5719 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500530 5719 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500534 5719 flags.go:64] FLAG: --max-pods="110" Oct 31 00:28:38.508370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500537 5719 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500540 5719 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500543 5719 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500545 5719 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500548 5719 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500551 5719 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500554 5719 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500560 5719 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500563 5719 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500568 5719 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500572 5719 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500575 5719 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500577 5719 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500580 5719 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500584 5719 flags.go:64] FLAG: --port="10250" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500587 5719 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500590 5719 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e6f2ba528be46ad4" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500593 5719 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500596 5719 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500599 5719 flags.go:64] FLAG: --register-node="true" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500602 5719 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500606 5719 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500609 5719 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500612 5719 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500614 5719 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500618 5719 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:28:38.509093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500621 5719 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500624 5719 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500627 5719 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500630 5719 flags.go:64] FLAG: --runonce="false" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500632 5719 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500636 5719 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500639 5719 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500641 5719 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500644 5719 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500647 5719 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500650 5719 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500653 5719 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500656 5719 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500658 5719 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500661 5719 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500665 5719 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500668 5719 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500671 5719 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500674 5719 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500679 5719 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500682 5719 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500685 5719 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500701 5719 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500705 5719 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500707 5719 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:28:38.509819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500711 5719 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500714 5719 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500717 5719 flags.go:64] FLAG: --v="2" Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500721 5719 flags.go:64] FLAG: --version="false" Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500726 5719 flags.go:64] FLAG: --vmodule="" Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500729 5719 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.500733 5719 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500858 5719 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500862 5719 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500865 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500868 5719 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500871 5719 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500874 5719 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500877 5719 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500879 5719 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500882 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500885 5719 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500888 5719 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500892 5719 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500895 5719 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500898 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:38.510730 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500900 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500904 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500907 5719 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500909 5719 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500912 5719 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500915 5719 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500917 5719 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500920 5719 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500922 5719 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500925 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500927 5719 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500931 5719 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500933 5719 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500936 5719 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500938 5719 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500941 5719 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500943 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500946 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500949 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:38.511239 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500952 5719 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500955 5719 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500957 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500959 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500962 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500964 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500968 5719 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500970 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500973 5719 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500975 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500977 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500980 5719 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500982 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500985 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500988 5719 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500990 5719 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500993 5719 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500995 5719 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.500998 5719 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:38.511792 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501000 5719 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501002 5719 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501005 5719 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501007 5719 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501010 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501013 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501016 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501018 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501021 5719 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501023 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501025 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501028 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501030 5719 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501033 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501035 5719 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501038 5719 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501040 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501042 5719 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501045 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:38.512463 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501047 5719 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501050 5719 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501053 5719 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501055 5719 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501058 5719 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501060 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501062 5719 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501065 5719 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501068 5719 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501070 5719 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501073 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501075 5719 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501077 5719 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501080 5719 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501082 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501085 5719 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501087 5719 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501090 5719 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501093 5719 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501096 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:38.512970 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501098 5719 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501101 5719 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501103 5719 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501106 5719 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501108 5719 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501111 5719 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501113 5719 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501116 5719 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501118 5719 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501121 5719 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501123 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501125 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501128 5719 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501130 5719 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501134 5719 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:38.513525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.501160 5719 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501194 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501198 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501201 5719 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501203 5719 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501206 5719 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501208 5719 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501211 5719 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501213 5719 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501216 5719 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501218 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501221 5719 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501224 5719 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501226 5719 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501229 5719 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501232 5719 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501235 5719 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501237 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501240 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:38.514021 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501242 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501245 5719 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501247 5719 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501250 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501252 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501255 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501257 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501260 5719 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501263 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501265 5719 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501267 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501270 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501272 5719 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501275 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501278 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501280 5719 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501283 5719 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501285 5719 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501287 5719 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501290 5719 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:38.514646 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501292 5719 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501295 5719 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501297 5719 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501299 5719 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501302 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501304 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501307 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501309 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501312 5719 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501315 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501317 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501320 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501322 5719 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501325 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501327 5719 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501330 5719 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501333 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501336 5719 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501338 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:38.515240 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501341 5719 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501343 5719 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501346 5719 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501349 5719 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501351 5719 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501353 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501356 5719 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501360 5719 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501362 5719 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501365 5719 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501367 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501370 5719 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501372 5719 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501375 5719 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501377 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501380 5719 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501382 5719 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501385 5719 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501387 5719 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501390 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:38.515803 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501393 5719 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501397 5719 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501399 5719 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501401 5719 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501405 5719 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501407 5719 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501410 5719 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501413 5719 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501415 5719 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501419 5719 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501422 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501424 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501427 5719 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501429 5719 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501433 5719 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501436 5719 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501439 5719 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501442 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501444 5719 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501446 5719 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:38.516302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501450 5719 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501452 5719 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501455 5719 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501457 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501459 5719 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501464 5719 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501467 5719 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501470 5719 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.501473 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.505227 5719 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.505241 5719 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.505260 5719 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.505264 5719 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505327 5719 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505334 5719 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505339 5719 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505344 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505348 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505353 5719 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:38.516807 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505358 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505362 5719 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505367 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505369 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505372 5719 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505374 5719 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505377 5719 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505380 5719 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505382 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505384 5719 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505387 5719 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505389 5719 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505392 5719 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505395 5719 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505398 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505401 5719 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505403 5719 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505406 5719 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505409 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505412 5719 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:38.517256 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505414 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505417 5719 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505420 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505422 5719 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505426 5719 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505428 5719 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505431 5719 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505435 5719 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505439 5719 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505445 5719 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505452 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505457 5719 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505462 5719 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505465 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505467 5719 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505470 5719 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505472 5719 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505475 5719 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505478 5719 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:38.517745 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505480 5719 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505482 5719 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505485 5719 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505488 5719 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505491 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505493 5719 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505496 5719 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505502 5719 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505507 5719 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505510 5719 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505512 5719 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505515 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505517 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505520 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505522 5719 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505526 5719 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505530 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505534 5719 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505538 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:38.518202 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505542 5719 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505546 5719 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505549 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505551 5719 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505554 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505557 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505559 5719 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505562 5719 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505564 5719 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505566 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505569 5719 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505572 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505575 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505577 5719 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505579 5719 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505582 5719 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505584 5719 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505587 5719 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505589 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505592 5719 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:38.518672 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505595 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505599 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505602 5719 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505606 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505610 5719 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505615 5719 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505619 5719 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505622 5719 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505625 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505627 5719 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505630 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505633 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505635 5719 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505638 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505640 5719 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505643 5719 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505645 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505648 5719 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505650 5719 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:38.519205 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505653 5719 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505655 5719 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.505658 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.505704 5719 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506588 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506604 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506609 5719 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506614 5719 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506619 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506624 5719 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506634 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506638 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506643 5719 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506647 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506651 5719 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506657 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:38.519703 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506662 5719 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506666 5719 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506670 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506674 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506678 5719 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506683 5719 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506687 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506713 5719 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506718 5719 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506722 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506726 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506730 5719 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506733 5719 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506738 5719 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506742 5719 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506746 5719 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506759 5719 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506763 5719 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506773 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:38.520144 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506778 5719 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506787 5719 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506793 5719 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506800 5719 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506804 5719 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506808 5719 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506812 5719 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506816 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506820 5719 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506824 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506829 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506839 5719 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506844 5719 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506865 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506883 5719 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506890 5719 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506894 5719 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506899 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506902 5719 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:38.520673 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506905 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506908 5719 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506911 5719 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506914 5719 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506917 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506920 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506927 5719 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506929 5719 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506932 5719 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506935 5719 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506938 5719 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506941 5719 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506944 5719 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506947 5719 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506949 5719 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506952 5719 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506955 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506957 5719 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506960 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.506965 5719 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:38.521184 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507001 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507006 5719 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507010 5719 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507013 5719 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507038 5719 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507042 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507143 5719 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507147 5719 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507150 5719 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507153 5719 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507156 5719 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507159 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507162 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507164 5719 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507167 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507169 5719 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507172 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507175 5719 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507177 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507180 5719 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:38.521676 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507183 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507185 5719 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507188 5719 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507191 5719 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507193 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507196 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507199 5719 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507201 5719 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507204 5719 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507206 5719 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507209 5719 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507211 5719 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507214 5719 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507216 5719 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507219 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507221 5719 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:38.522203 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.507257 5719 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507297 5719 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507304 5719 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507308 5719 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507311 5719 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507313 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507316 5719 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507319 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507322 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507325 5719 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507328 5719 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507330 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507334 5719 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507338 5719 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507341 5719 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507343 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507346 5719 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507348 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507351 5719 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:28:38.522627 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507354 5719 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507357 5719 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507360 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507362 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507365 5719 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507367 5719 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507370 5719 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507372 5719 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507375 5719 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507378 5719 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507380 5719 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507383 5719 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507385 5719 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507388 5719 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507390 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507393 5719 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507395 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507398 5719 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507400 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507402 5719 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:28:38.523119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507405 5719 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507407 5719 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507410 5719 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507413 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507416 5719 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507419 5719 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507421 5719 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507424 5719 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507426 5719 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507429 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507434 5719 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507437 5719 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507439 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507442 5719 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507444 5719 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507447 5719 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507449 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507453 5719 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507456 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507459 5719 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:28:38.523657 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507461 5719 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507463 5719 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507466 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507469 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507471 5719 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507474 5719 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507476 5719 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507478 5719 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507481 5719 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507483 5719 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507486 5719 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507488 5719 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507491 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507493 5719 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507496 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507498 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507501 5719 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507504 5719 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507507 5719 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507509 5719 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:28:38.524192 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507512 5719 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507514 5719 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507518 5719 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507520 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507523 5719 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507527 5719 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507529 5719 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507532 5719 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507534 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507537 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507539 5719 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507542 5719 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507544 5719 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507547 5719 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507549 5719 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507552 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507554 5719 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507556 5719 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507559 5719 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:28:38.524669 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507562 5719 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507564 5719 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507567 5719 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507569 5719 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507572 5719 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507574 5719 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507577 5719 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507579 5719 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:28:38.507584 5719 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.507642 5719 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.508627 5719 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.508677 5719 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.509049 5719 server.go:1014] "Starting client certificate rotation" Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.509191 5719 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.509240 5719 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:22:25 +0000 UTC" deadline="2028-04-28 02:02:15.372250085 +0000 UTC" Oct 31 00:28:38.525310 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.509267 5719 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13081h33m36.86298506s" Oct 31 00:28:38.526009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.509512 5719 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:28:38.526009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.510661 5719 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:28:38.526009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.511959 5719 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:28:38.526009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.513983 5719 log.go:25] "Validated CRI v1 image API" Oct 31 00:28:38.526009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.514709 5719 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:28:38.526009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.516895 5719 fs.go:135] Filesystem UUIDs: map[462dfc59-2b59-4aaa-be6c-d90aeee357e0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fe15e7c3-57af-42d9-92d2-22ef711010e7:/dev/nvme0n1p3] Oct 31 00:28:38.526190 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.516916 5719 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16f55a4d9dc60a00d646c17530e5b9a9468c4e2cc45988cace33d484ca02c6fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16f55a4d9dc60a00d646c17530e5b9a9468c4e2cc45988cace33d484ca02c6fb/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ce35bba90356bc4bfffbc2963fe77982ba0636d6e5fa8e8d0db05461ae0bded/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ce35bba90356bc4bfffbc2963fe77982ba0636d6e5fa8e8d0db05461ae0bded/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d02da165f9aba2e1411d24cfc79250d0608eacb72f3fb742edc1e6ddb9fe517/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d02da165f9aba2e1411d24cfc79250d0608eacb72f3fb742edc1e6ddb9fe517/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e1473fbc5a0aa9a10f7af5ce005b8c5ba302e4550e8bc369a056f768da5646e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e1473fbc5a0aa9a10f7af5ce005b8c5ba302e4550e8bc369a056f768da5646e/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2633f481618d7b185b582b1473ccf23b696e7ea663d8656b96ebed351c37134b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2633f481618d7b185b582b1473ccf23b696e7ea663d8656b96ebed351c37134b/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b9f62e5b02c9a2156d4509505e6023213fc1cb28842032fef23adfcdbe50853/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b9f62e5b02c9a2156d4509505e6023213fc1cb28842032fef23adfcdbe50853/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/325df31cc31fd7f9e658af6a13c49330aecddac9cd4f4b550230fe96f2f0a7af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/325df31cc31fd7f9e658af6a13c49330aecddac9cd4f4b550230fe96f2f0a7af/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42730e787618364ebf636ece433affc2dec265043991ca6c9540f88da6b2fa6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42730e787618364ebf636ece433affc2dec265043991ca6c9540f88da6b2fa6a/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ddd7ba8d34be7c0899e8c77bb5d0bd116657b2b0bf891271a6147ae1fbe39a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ddd7ba8d34be7c0899e8c77bb5d0bd116657b2b0bf891271a6147ae1fbe39a8/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bf5130065b585a73bd9710561277316c817d70e2932411408d85f8863430b3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bf5130065b585a73bd9710561277316c817d70e2932411408d85f8863430b3c/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67e9aa3626c4c2b2c7908325b4067ccf200a80786073f9ba611fb17a551c986c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67e9aa3626c4c2b2c7908325b4067ccf200a80786073f9ba611fb17a551c986c/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71c7e6739da357f0d3b4841383108c16d6dd1e3a85ea29a4ed29e866439d1053/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71c7e6739da357f0d3b4841383108c16d6dd1e3a85ea29a4ed29e866439d1053/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74e42a6ef1d8accc3c3df3bbe9c8a6c41da448a0619a1933337b9d406b0e1844/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74e42a6ef1d8accc3c3df3bbe9c8a6c41da448a0619a1933337b9d406b0e1844/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91142f3525707602388be1bccce15518cceb9860c4cf2c39f4243cdea857d465/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91142f3525707602388be1bccce15518cceb9860c4cf2c39f4243cdea857d465/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93567fb71b263cb3b775692c7c043e352bf3e90c5255a676ac23d752ca02ecc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93567fb71b263cb3b775692c7c043e352bf3e90c5255a676ac23d752ca02ecc6/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/957b2fe6a5227d581b085f387356ffaf5ac37ba89d6d5abe6fe6eb147c39db86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/957b2fe6a5227d581b085f387356ffaf5ac37ba89d6d5abe6fe6eb147c39db86/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95cc97cc04fa989cc8f45aa3e76a34fd6778bb5107d5465644662c66f6c88c80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95cc97cc04fa989cc8f45aa3e76a34fd6778bb5107d5465644662c66f6c88c80/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5854abe3749dd55c8aeb07c3d7894ed8a35e88d2fa28f56418efb86b5b309fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5854abe3749dd55c8aeb07c3d7894ed8a35e88d2fa28f56418efb86b5b309fd/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/adf94de2be2239c94b6926e78a9413109c3c9e0e7ab2706bfa14317941079985/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/adf94de2be2239c94b6926e78a9413109c3c9e0e7ab2706bfa14317941079985/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c207d12fda7f63e8b83e7ad118e1da170cd71ff8b630ad64f7616d0502cd827f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c207d12fda7f63e8b83e7ad118e1da170cd71ff8b630ad64f7616d0502cd827f/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d48dfd49ce4b1e5cd6d20bfa6e1d12ee1f821af1f6a7f8b8d21a279656cca0fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d48dfd49ce4b1e5cd6d20bfa6e1d12ee1f821af1f6a7f8b8d21a279656cca0fd/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/daa5e7ad4eee821437d8f097c3b127c37c84d43fc2333f1f50dbc7a8e6fb6830/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/daa5e7ad4eee821437d8f097c3b127c37c84d43fc2333f1f50dbc7a8e6fb6830/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc90f34210244e8bc0b66d0db9eaf95da4c4de912a114d9ff3702d4854014d65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc90f34210244e8bc0b66d0db9eaf95da4c4de912a114d9ff3702d4854014d65/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e059d0603b0e71ebf91d721d4911a8d57b734f52c1589e2e179a8aef6b367802/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e059d0603b0e71ebf91d721d4911a8d57b734f52c1589e2e179a8aef6b367802/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3d8a2f581559afaed0caeb6c409eeaf66a72fd367483fa96e94a6649fadb40b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3d8a2f581559afaed0caeb6c409eeaf66a72fd367483fa96e94a6649fadb40b/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d861409-3dd5-4eed-ada7-e1817b7fb58f/volumes/kubernetes.io~projected/kube-api-access-trktv:{mountpoint:/var/lib/kubelet/pods/0d861409-3dd5-4eed-ada7-e1817b7fb58f/volumes/kubernetes.io~projected/kube-api-access-trktv major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d861409-3dd5-4eed-ada7-e1817b7fb58f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0d861409-3dd5-4eed-ada7-e1817b7fb58f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~projected/kube-api-access-tcd45:{mountpoint:/var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~projected/kube-api-access-tcd45 major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~projected/kube-api-access-p4gq2:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~projected/kube-api-access-p4gq2 major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~projected/tls-assets major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/config major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/web-config major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~projected/kube-api-access-klpkz:{mountpoint:/var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~projected/kube-api-access-klpkz major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16395bf6-131d-4558-8bfe-83d994039370/volumes/kubernetes.io~projected/kube-api-access-95dhq:{mountpoint:/var/lib/kubelet/pods/16395bf6-131d-4558-8bfe-83d994039370/volumes/kubernetes.io~projected/kube-api-access-95dhq major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16395bf6-131d-4558-8bfe-83d994039370/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/16395bf6-131d-4558-8bfe-83d994039370/volumes/kubernetes.io~secret/metrics-tls major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/204028bd-7b66-4354-bd39-54aaca835224/volumes/kubernetes.io~projected/kube-api-access-dz6mf:{mountpoint:/var/lib/kubelet/pods/204028bd-7b66-4354-bd39-54aaca835224/volumes/kubernetes.io~projected/kube-api-access-dz6mf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/204028bd-7b66-4354-bd39-54aaca835224/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/204028bd-7b66-4354-bd39-54aaca835224/volumes/kubernetes.io~secret/cert major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5476b645-f064-41af-b1da-9f7644f281d0/volumes/kubernetes.io~projected/kube-api-access-chbng:{mountpoint:/var/lib/kubelet/pods/5476b645-f064-41af-b1da-9f7644f281d0/volumes/kubernetes.io~projected/kube-api-access-chbng major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~projected/kube-api-access-h7htb:{mountpoint:/var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~projected/kube-api-access-h7htb major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64a235e0-e510-479f-a6bd-92cc4910a1e5/volumes/kubernetes.io~projected/kube-api-access-qk75x:{mountpoint:/var/lib/kubelet/pods/64a235e0-e510-479f-a6bd-92cc4910a1e5/volumes/kubernetes.io~projected/kube-api-access-qk75x major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/alertmanager/9 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/config-reloader/6 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~projected/kube-api-access-56zs9:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~projected/kube-api-access-56zs9 major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~projected/tls-assets major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/config-volume major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/web-config major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~projected/kube-api-access-4p2mc:{mountpoint:/var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~projected/kube-api-access-4p2mc major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9dc277-9345-44d1-9f7e-90b5a9f3380e/volumes/kubernetes.io~projected/kube-api-access-f87bx:{mountpoint:/var/lib/kubelet/pods/8b9dc277-9345-44d1-9f7e-90b5a9f3380e/volumes/kubernetes.io~projected/kube-api-access-f87bx major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/963fd145-6705-467a-bac7-25b3ff8add39/volumes/kubernetes.io~projected/kube-api-access-dpqq8:{mountpoint:/var/lib/kubelet/pods/963fd145-6705-467a-bac7-25b3ff8add39/volumes/kubernetes.io~projected/kube-api-access-dpqq8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9d50249-2496-4219-827f-72c9043805fe/volumes/kubernetes.io~projected/kube-api-access-vnmsw:{mountpoint:/var/lib/kubelet/pods/a9d50249-2496-4219-827f-72c9043805fe/volumes/kubernetes.io~projected/kube-api-access-vnmsw major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9d50249-2496-4219-827f-72c9043805fe/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a9d50249-2496-4219-827f-72c9043805fe/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aad04f16-056a-4115-97a0-e75d9e8b36b7/volumes/kubernetes.io~projected/kube-api-access-5z6vv:{mountpoint:/var/lib/kubelet/pods/aad04f16-056a-4115-97a0-e75d9e8b36b7/volumes/kubernetes.io~projected/kube-api-access-5z6vv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1822938-c344-40c9-99c1-986663748169/volumes/kubernetes.io~projected/kube-api-access-rqx6g:{mountpoint:/var/lib/kubelet/pods/b1822938-c344-40c9-99c1-986663748169/volumes/kubernetes.io~projected/kube-api-access-rqx6g major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volumes/kubernetes.io~projected/kube-api-access-ngbqj:{mountpoint:/var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volumes/kubernetes.io~projected/kube-api-access-ngbqj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b75cde1b-aaee-46c8-b956-dcecf3b6afbf/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b75cde1b-aaee-46c8-b956-dcecf3b6afbf/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~projected/kube-api-access-2qsl4:{mountpoint:/var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~projected/kube-api-access-2qsl4 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~projected/kube-api-access-q5k99:{mountpoint:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~projected/kube-api-access-q5k99 major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~projected/kube-api-access-hg94p:{mountpoint:/var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~projected/kube-api-access-hg94p major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6d7d3bf-205d-4251-9b47-400892684e0e/volumes/kubernetes.io~projected/kube-api-access-6krdc:{mountpoint:/var/lib/kubelet/pods/d6d7d3bf-205d-4251-9b47-400892684e0e/volumes/kubernetes.io~projected/kube-api-access-6krdc major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fda3848b-9d0b-4832-8789-862920d484b8/volumes/kubernetes.io~projected/kube-api-access-vdd6j:{mountpoint:/var/lib/kubelet/pods/fda3848b-9d0b-4832-8789-862920d484b8/volumes/kubernetes.io~projected/kube-api-access-vdd6j major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fda3848b-9d0b-4832-8789-862920d484b8/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/fda3848b-9d0b-4832-8789-862920d484b8/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:360 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/45cece97297fd9f0b51f007d329d2a2f91d95d1b8946329535c8633b459f16aa/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d376bb67b09cb700107eb85c3d0a7913a422ad89c3995190cb0151c0d814d420/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f243670cdedad5da2c05d655dc0cc00478aeefae91e3a781d8d2011fb680d8c2/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/aca1003ffacb4c7c1756675f4e309c8dbe73b0e2124958941e755a0a1ce4771a/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/fe68a4d75ec6b4b6ec0fc448d22ef83c0e613dd933c9a4e9a6370173e0ebc4c1/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/5f7b27b31a9f75d66827b28d78e4c84f6ee5d71aee8b483a30fd2dade60fdbd1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/10ae2269345ff9eed96ef6c14fb870c652ea6573f62ecdc15f718c6bbb8d45c5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/55b954b160fef34a341bb050410d87fea14b1a5f2480e640ea39e225484095ee/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/067cee7e535a1ad93eecbb03d90576f09fbe25a1fc3d44ed70b50cefeb33fcab/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/3f71dfacc14366c65ed320912d9796621ff3ffe552bb63094e99535368605eab/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/4bfc5e8d851adfa6a143f7caab7b20d95c1e9fb8d67d6558d8b5c19ce12203ee/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/7c7700fa3d54255a507e5ac58716b527b36d5126ef0594d2e474c1711c9ae734/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/89aa82710c5a7c5c36528aa2170c2dc991def4c361e525d7a9d622eb5fdcc557/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/6c4d92aecc0ce1aa3f17fb69b6924867988acdb229561e00e6a42a6f8be92f7d/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/36331fbe1fb9f82cad34bd99af72fd4fbd97e220798b1f3a7a71c72d5cc64bc0/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/034ba19b1e7cfd9177dc993f24e08de0d2a0bb78a08cde3b5e2bd21dcc8bfb1c/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/56973a465a7dba9f46e6ff6e95e2de01cf59f78d865724ae548da7545577212a/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/bc2ceae918267bd25dbff62243ffa6eb1b0eaa23ad48e046ec90e97d1a306a4f/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/26de3917692490e7604039964d8bc0ea0ce71eaa9a49512b66d280d6a1accc07/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/56437bbbb1bcb44452d9143ce51924b64dc55008fa4db8b3d7b0e8cd899743da/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/54303675b5f9cb64193b08d521f849e4a5c9bbeea52a65a25d1b82008ac755d3/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e2d472f0c9c361b7849feef0492e07e1a52036680a9dfe5087edb95e4b1f174a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/95d0ec86f9ac2d3529a5ec0e7885d4aae7e436cd3150553996b46a810556bbc3/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ff7fb1a975bcfe05803784d359b019db7c51a521ae7424fe44cc44347baff9c7/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/d2007ea8a5f4fe56763068fc27e175522de2cc0bd3aa57f89af423080fedca07/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/59c05ea00dba2fcce42ae68d6df0a1c10a66fac09d964eb1d595fad4f4376cd0/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/648546fd249758bd2d646d860fc388d68f2d43e178526836bcbea49151cdaba2/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/69912652f8cef974aed2fdba8d8b3413727cb26b82d54c9dff0217a80d038423/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/efa6dbb7bef1ef20bef2c58d39540fe6a0600f53da5b1e0c53ad8582f75fdffe/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/a28e12078ef0341aea3723a663f4e948c28119a5af16d3e21609a16a18ddfc91/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/8a9a4bb69bf24d49580f5a5f9c0efeef4f3942f1489d6f5f10bb76c89e91358f/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/5c56cfff7fd4f345afb35fd79d0a1be24426d2c2e0a7daf9a242917c26a302ea/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/efe61dd0ce4542b743cc1c4c39d535efa39a915f7ce5eb3311a6f829c3e63dc5/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/2e602396e8775345a000a9911d376d59e4e96e6f1360619edf5b92421983eabf/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/12f22e118543145e9e6b428d10e31c4f130e0fdc3ddc55d5a470f062676d152f/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/81c1285e6bffbcc4934394ea6d17482819c8f965e8123c943564849f2b8b07fa/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/fc85e330b62fef5c4614c1a4894961dbe0aec7549f187cb7c64a437aabdb9a65/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/6b63ee1eb951658d23cdb6ba0d998c31b4eea3f27b1f52db1676c3943f5300d6/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/96dd5a95c46baa59148fbe8bb3145ca2657da746ab5bbd61fde5ce9ac6df2a93/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/f495235091adc0369f3d3611ed6cd734038800e5357508d96867ba9de1589e9d/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/3252aec5fcc89d8250730e41e69184115a34aa4b9b5bd7027862bd5a51e350f9/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/1a9a4d0f87c392170bf36690ad9e999edfde6e40deaf09e6d49e4a4a2f5eb642/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/624c4bcb3dd03ab7baadc9fe0c793627172b7b5e5afc70559d6d7053abbe50c8/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/7302c67f0148177f441a8c04cc4012ba042b91af9384eb98c00ceb23767d5de2/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/95ed569e4058e60d665d5fd2e7830a1dcf635e0230d3d420ff06e3091c2d735b/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/16777fa5e44fd593420cd8ae30d7cd3a59b4312d19758c8f6cb1c2d9b20b1625/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/a6a3d7c30459ffb5a10aaf82a570e9170d197928b8d37d8b41048db61a7f69f9/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/51b75cadaeb0eb00dee59861eb3eac8fa228b19c34e95811a7a98537c8c055f9/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/95ce64b677fe04b4dfbaa07ced0707cc1a12c5ef1620c6d27d548f4a45d6ce28/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/56111bdd350781cc4987426597c515343b18bf353496fcd7de77a5ce5fe874ca/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/65b052b7267ccdbf5b103ced90b41f7a38c563b6bf26455cd6f58bf3025fba5d/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/4d2c08def1fef0dd8b59baef5fef87b199eabd8de24a1e2ad1ff14a54fd6cd64/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/0c90ae120b11f23b008b966082b85ebc7cba75971407f4c94cdac24a3d8cc343/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/f23abff1889e848ab09b7490734733e382d79f30f8af2c9b16bfcbdd313cb955/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/769fe73a537617dcf09ee36dbfca30077671a72917476a48482de11f6812b1f7/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/88bf1ea299b4e0f37cc7a8ff4408eea39d85263b8d969c6974c05a688d67d971/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/1e04caf547c7f012ef1fbc9d9cc4c39da02566bedd8078b675cdc8c662e031ca/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/d148fdeacba47a2085fc8c358d794867d346886d802878af6d98db7b3ac4cbb9/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/fd5d82760b091d846a505a85c690d02ee1fc3ab928526b3b29be2f7127bee621/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/3b30a193326f6a4e72b26cd61c95ab60c70e5ac890b692b3c65e94d243aac308/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/303d0cd7a03939fc95698b803b5fc72d40ca392cabde17422edd5515527b799b/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/a66f342624632c27d95ba86947d5aeb550fb4eb49bb385977a8015f4667b28de/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/5c744fc5ce98c8bab769a32c64a60bf107eeab01f0a41d5f0b9a701faca86fba/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/e4520f83224046dc910a189d4e0871dff6a160c088bd79d9676a2fd11df805d7/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/270fc52c3abc7c6d0bf27051383bc687bb13001af2824a5890839ad0d4bfb80a/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/79dad7a209c679657699915eb06d8baf13b3932bf362fc7a43b2a021837b3935/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/749e8a56ba861bd18618a48aa2538443ba24478771c540aade00316a92efbafd/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-479:{mountpoint:/var/lib/containers/storage/overlay/34b778d39897c795e7f24cb1511c547c4d2706bbb754a7f0247bccac2596c0b1/merged major:0 minor:479 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/fd6cd096f48ef4ef645e1fa9cde32e9cee6e8b6aeb1ec0d24260baf554f0678c/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/2522aae90392050469546b9efbdfd99880860a6d11a4b9a46eb68113f6e757c4/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/b2902a8a76811ee493f046e501e8564d50ebd20efbc575f7ab93ad21aedcdd42/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6b3078fd6cc6ce5fdf449dd7537e0a517617999aca46d4cb44146b0963f819d3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4ff5d74398672c53648bc5023dc0f3d4b26dab607b86beb517dfbd4567159475/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/15c9e64e443e9981cd1d58d7fa50ef5f98951081770934ebe124edb738c3254c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/5546c5e958d123ef60f10e9b2220b160d5ff9ccb297cb48365fc6760efcacf56/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/473330ef4937a8bfef5a9f6770dde21a753aa85df3d2a15d5635a59d44b3f652/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/1f2f8bf016e30e2a25154c523eea63a07f575fc017ced8a8678759ddcb9479a7/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/350b0826598dcc8412274586849729e3185fdb4726549d3aac6c938de1a8ca02/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/5e1928505615a91e938e47325102d0f04d165ed6b37ca1ff899862ad9d1e9b1e/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 31 00:28:38.527971 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.527502 5719 manager.go:217] Machine: {Timestamp:2026-10-31 00:28:38.527040873 +0000 UTC m=+0.063183168 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092280 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f8aefa33f1e6ba02501df9a029d8 SystemUUID:ec25f8ae-fa33-f1e6-ba02-501df9a029d8 BootID:ead03ab6-5c5f-4995-9c3f-8c8102a41b81 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc90f34210244e8bc0b66d0db9eaf95da4c4de912a114d9ff3702d4854014d65/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:327 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e1473fbc5a0aa9a10f7af5ce005b8c5ba302e4550e8bc369a056f768da5646e/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~projected/kube-api-access-tcd45 DeviceMajor:0 DeviceMinor:252 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:326 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:420 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:429 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ddd7ba8d34be7c0899e8c77bb5d0bd116657b2b0bf891271a6147ae1fbe39a8/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74e42a6ef1d8accc3c3df3bbe9c8a6c41da448a0619a1933337b9d406b0e1844/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~projected/kube-api-access-h7htb DeviceMajor:0 DeviceMinor:365 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:425 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a9d50249-2496-4219-827f-72c9043805fe/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/963fd145-6705-467a-bac7-25b3ff8add39/volumes/kubernetes.io~projected/kube-api-access-dpqq8 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/204028bd-7b66-4354-bd39-54aaca835224/volumes/kubernetes.io~projected/kube-api-access-dz6mf DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-479 DeviceMajor:0 DeviceMinor:479 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d6d7d3bf-205d-4251-9b47-400892684e0e/volumes/kubernetes.io~projected/kube-api-access-6krdc DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~projected/kube-api-access-56zs9 DeviceMajor:0 DeviceMinor:291 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:424 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:320 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d861409-3dd5-4eed-ada7-e1817b7fb58f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:321 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:362 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:257 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:296 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:428 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a9d50249-2496-4219-827f-72c9043805fe/volumes/kubernetes.io~projected/kube-api-access-vnmsw DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~projected/kube-api-access-4p2mc DeviceMajor:0 DeviceMinor:260 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:301 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0284bc6-6a9b-42ed-b443-1f94367c8ac4/volumes/kubernetes.io~projected/kube-api-access-2qsl4 DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e059d0603b0e71ebf91d721d4911a8d57b734f52c1589e2e179a8aef6b367802/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:279 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:303 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/325df31cc31fd7f9e658af6a13c49330aecddac9cd4f4b550230fe96f2f0a7af/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95cc97cc04fa989cc8f45aa3e76a34fd6778bb5107d5465644662c66f6c88c80/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/adf94de2be2239c94b6926e78a9413109c3c9e0e7ab2706bfa14317941079985/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93567fb71b263cb3b775692c7c043e352bf3e90c5255a676ac23d752ca02ecc6/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~projected/kube-api-access-q5k99 DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:303 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b75cde1b-aaee-46c8-b956-dcecf3b6afbf/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2633f481618d7b185b582b1473ccf23b696e7ea663d8656b96ebed351c37134b/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:256 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c207d12fda7f63e8b83e7ad118e1da170cd71ff8b630ad64f7616d0502cd827f/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:363 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16f55a4d9dc60a00d646c17530e5b9a9468c4e2cc45988cace33d484ca02c6fb/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/daa5e7ad4eee821437d8f097c3b127c37c84d43fc2333f1f50dbc7a8e6fb6830/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f65619e-5f82-4313-8a35-572fbece3755/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:364 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:303 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5476b645-f064-41af-b1da-9f7644f281d0/volumes/kubernetes.io~projected/kube-api-access-chbng DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:421 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5854abe3749dd55c8aeb07c3d7894ed8a35e88d2fa28f56418efb86b5b309fd/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:295 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67e9aa3626c4c2b2c7908325b4067ccf200a80786073f9ba611fb17a551c986c/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:423 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~projected/kube-api-access-p4gq2 DeviceMajor:0 DeviceMinor:432 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64a235e0-e510-479f-a6bd-92cc4910a1e5/volumes/kubernetes.io~projected/kube-api-access-qk75x DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/16395bf6-131d-4558-8bfe-83d994039370/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:196 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:297 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ce35bba90356bc4bfffbc2963fe77982ba0636d6e5fa8e8d0db05461ae0bded/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:298 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:302 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:300 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volumes/kubernetes.io~projected/kube-api-access-ngbqj DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9dc277-9345-44d1-9f7e-90b5a9f3380e/volumes/kubernetes.io~projected/kube-api-access-f87bx DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fda3848b-9d0b-4832-8789-862920d484b8/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:360 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fda3848b-9d0b-4832-8789-862920d484b8/volumes/kubernetes.io~projected/kube-api-access-vdd6j DeviceMajor:0 DeviceMinor:361 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:416 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/204028bd-7b66-4354-bd39-54aaca835224/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:200 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:420 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71c7e6739da357f0d3b4841383108c16d6dd1e3a85ea29a4ed29e866439d1053/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d02da165f9aba2e1411d24cfc79250d0608eacb72f3fb742edc1e6ddb9fe517/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42730e787618364ebf636ece433affc2dec265043991ca6c9540f88da6b2fa6a/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/b42b8ff9-0140-4b1c-97f2-131bc141352f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d861409-3dd5-4eed-ada7-e1817b7fb58f/volumes/kubernetes.io~projected/kube-api-access-trktv DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:427 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3d8a2f581559afaed0caeb6c409eeaf66a72fd367483fa96e94a6649fadb40b/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:319 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d0910c96-18d0-49b2-861d-0ec2a5793047/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:325 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:422 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/aad04f16-056a-4115-97a0-e75d9e8b36b7/volumes/kubernetes.io~projected/kube-api-access-5z6vv DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b1822938-c344-40c9-99c1-986663748169/volumes/kubernetes.io~projected/kube-api-access-rqx6g DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b9f62e5b02c9a2156d4509505e6023213fc1cb28842032fef23adfcdbe50853/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:426 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:303 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:301 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d48dfd49ce4b1e5cd6d20bfa6e1d12ee1f821af1f6a7f8b8d21a279656cca0fd/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/957b2fe6a5227d581b085f387356ffaf5ac37ba89d6d5abe6fe6eb147c39db86/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bf5130065b585a73bd9710561277316c817d70e2932411408d85f8863430b3c/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d142f31b-3461-4f5a-93e1-088ce6511100/volumes/kubernetes.io~projected/kube-api-access-hg94p DeviceMajor:0 DeviceMinor:259 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes/kubernetes.io~projected/kube-api-access-klpkz DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:299 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/16395bf6-131d-4558-8bfe-83d994039370/volumes/kubernetes.io~projected/kube-api-access-95dhq DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91142f3525707602388be1bccce15518cceb9860c4cf2c39f4243cdea857d465/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16f55a4d9dc60a0 MacAddress:26:85:86:f8:b1:b1 Speed:10000 Mtu:8901} {Name:1ce35bba90356bc MacAddress:c6:8f:7a:93:61:30 Speed:10000 Mtu:8901} {Name:1e1473fbc5a0aa9 MacAddress:ea:4f:32:e8:6e:76 Speed:10000 Mtu:8901} {Name:2633f481618d7b1 MacAddress:5e:96:2c:96:ee:dd Speed:10000 Mtu:8901} {Name:2b9f62e5b02c9a2 MacAddress:6e:18:e2:c7:2a:07 Speed:10000 Mtu:8901} {Name:67e9aa3626c4c2b MacAddress:62:27:70:c0:31:b2 Speed:10000 Mtu:8901} {Name:71c7e6739da357f MacAddress:4a:d1:32:ee:a2:2b Speed:10000 Mtu:8901} {Name:91142f352570760 MacAddress:6a:c4:ef:14:c9:7e Speed:10000 Mtu:8901} {Name:95cc97cc04fa989 MacAddress:fe:24:4f:5b:31:44 Speed:10000 Mtu:8901} {Name:a5854abe3749dd5 MacAddress:2e:4e:9a:94:7a:ba Speed:10000 Mtu:8901} {Name:adf94de2be2239c MacAddress:8a:24:7e:a8:b5:f3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ec:c8:1b:85 Speed:0 Mtu:9001} {Name:br-int MacAddress:52:fb:da:88:a9:a1 Speed:0 Mtu:8901} {Name:c207d12fda7f63e MacAddress:a2:6e:6e:51:3b:e9 Speed:10000 Mtu:8901} {Name:d48dfd49ce4b1e5 MacAddress:26:f6:ec:5d:7f:49 Speed:10000 Mtu:8901} {Name:daa5e7ad4eee821 MacAddress:66:37:12:50:38:b3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:ec:c8:1b:85 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:b5:08:4b:df:65 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:02:ff:bb:89:4a:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:28:38.527971 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.527976 5719 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:28:38.528321 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528028 5719 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:28:38.528375 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528341 5719 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:28:38.528498 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528379 5719 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-8.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:28:38.528540 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528500 5719 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:28:38.528540 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528508 5719 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:28:38.528540 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528521 5719 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:28:38.528613 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528540 5719 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:28:38.528646 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528614 5719 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:28:38.528785 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528774 5719 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:28:38.528877 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528869 5719 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:28:38.528913 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528902 5719 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:28:38.528942 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528915 5719 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:28:38.528942 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528925 5719 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:28:38.528942 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528934 5719 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:28:38.528942 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.528941 5719 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:28:38.529560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.529545 5719 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:28:38.529560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.529557 5719 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:28:38.530040 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.530013 5719 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:28:38.552230 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552210 5719 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:28:38.552230 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552225 5719 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:28:38.552349 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552338 5719 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:28:38.552530 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552518 5719 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:28:38.552560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552549 5719 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:28:38.552656 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552647 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:28:38.552685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552663 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:28:38.552685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552669 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:28:38.552685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552676 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:28:38.552685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552686 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552712 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552720 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552726 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552732 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552738 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552753 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552762 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552792 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:28:38.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.552801 5719 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:28:38.556109 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.556096 5719 server.go:1263] "Started kubelet" Oct 31 00:28:38.556196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.556142 5719 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:28:38.556615 ip-10-0-129-8 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:28:38.557557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.556196 5719 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:28:38.557557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.557028 5719 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:28:38.557557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.557347 5719 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:28:38.557739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.557728 5719 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:28:38.563003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.562402 5719 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:28:38.563003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.562421 5719 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:28:38.563003 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.562469 5719 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-8.us-west-2.compute.internal\" not found" Oct 31 00:28:38.563003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.562487 5719 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:28:38.563003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.562479 5719 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:28:38.563003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.562477 5719 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:22:26 +0000 UTC" deadline="2028-05-22 11:56:17.03175825 +0000 UTC" Oct 31 00:28:38.563003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.562513 5719 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13667h27m38.469248003s" Oct 31 00:28:38.563003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.562505 5719 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:28:38.564511 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.564384 5719 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:28:38.564511 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.564401 5719 factory.go:55] Registering systemd factory Oct 31 00:28:38.564511 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.564409 5719 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:28:38.565712 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565664 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 00:28:38.565793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565710 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75963a1c-a5af-4c12-8adc-f6c5b9aa2e39" volumeName="kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 00:28:38.565793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565732 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 00:28:38.565793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565744 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d861409-3dd5-4eed-ada7-e1817b7fb58f" volumeName="kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs" seLinuxMountContext="" Oct 31 00:28:38.565793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565755 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 00:28:38.565793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565777 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets" seLinuxMountContext="" Oct 31 00:28:38.565793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565788 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16395bf6-131d-4558-8bfe-83d994039370" volumeName="kubernetes.io/secret/16395bf6-131d-4558-8bfe-83d994039370-metrics-tls" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565799 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5" volumeName="kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565816 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b42b8ff9-0140-4b1c-97f2-131bc141352f" volumeName="kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-config" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565831 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d142f31b-3461-4f5a-93e1-088ce6511100" volumeName="kubernetes.io/empty-dir/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-textfile" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565842 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565859 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368083a-8041-45bb-8e12-93f10e05eb15" volumeName="kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565878 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565891 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368083a-8041-45bb-8e12-93f10e05eb15" volumeName="kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565912 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b42b8ff9-0140-4b1c-97f2-131bc141352f" volumeName="kubernetes.io/secret/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565922 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/projected/d0910c96-18d0-49b2-861d-0ec2a5793047-kube-api-access-q5k99" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565934 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565949 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368083a-8041-45bb-8e12-93f10e05eb15" volumeName="kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565960 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.565999 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75963a1c-a5af-4c12-8adc-f6c5b9aa2e39" volumeName="kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-metrics-client-ca" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566016 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9d50249-2496-4219-827f-72c9043805fe" volumeName="kubernetes.io/configmap/a9d50249-2496-4219-827f-72c9043805fe-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566028 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1822938-c344-40c9-99c1-986663748169" volumeName="kubernetes.io/empty-dir/b1822938-c344-40c9-99c1-986663748169-tmp-dir" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566043 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566062 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566083 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75963a1c-a5af-4c12-8adc-f6c5b9aa2e39" volumeName="kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566095 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:28:38.566103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566110 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566121 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566136 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75963a1c-a5af-4c12-8adc-f6c5b9aa2e39" volumeName="kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566156 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b42b8ff9-0140-4b1c-97f2-131bc141352f" volumeName="kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566171 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368083a-8041-45bb-8e12-93f10e05eb15" volumeName="kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566182 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5476b645-f064-41af-b1da-9f7644f281d0" volumeName="kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566198 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f65619e-5f82-4313-8a35-572fbece3755" volumeName="kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-client-ca-bundle" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566208 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9d50249-2496-4219-827f-72c9043805fe" volumeName="kubernetes.io/secret/a9d50249-2496-4219-827f-72c9043805fe-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566219 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566234 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/empty-dir/d0910c96-18d0-49b2-861d-0ec2a5793047-tmp" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566245 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566262 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16395bf6-131d-4558-8bfe-83d994039370" volumeName="kubernetes.io/projected/16395bf6-131d-4558-8bfe-83d994039370-kube-api-access-95dhq" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566319 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5476b645-f064-41af-b1da-9f7644f281d0" volumeName="kubernetes.io/projected/5476b645-f064-41af-b1da-9f7644f281d0-kube-api-access-chbng" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566337 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad04f16-056a-4115-97a0-e75d9e8b36b7" volumeName="kubernetes.io/projected/aad04f16-056a-4115-97a0-e75d9e8b36b7-kube-api-access-5z6vv" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566347 5719 factory.go:153] Registering CRI-O factory Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566364 5719 factory.go:223] Registration of the crio container factory successfully Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566355 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/configmap/d0910c96-18d0-49b2-861d-0ec2a5793047-metrics-client-ca" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566383 5719 factory.go:103] Registering Raw factory Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566404 5719 manager.go:1199] Started watching for new ooms in manager Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566406 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566420 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566434 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9dc277-9345-44d1-9f7e-90b5a9f3380e" volumeName="kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566467 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9d50249-2496-4219-827f-72c9043805fe" volumeName="kubernetes.io/empty-dir/a9d50249-2496-4219-827f-72c9043805fe-data-volume" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566523 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75cde1b-aaee-46c8-b956-dcecf3b6afbf" volumeName="kubernetes.io/secret/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-agent-certs" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566538 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75963a1c-a5af-4c12-8adc-f6c5b9aa2e39" volumeName="kubernetes.io/projected/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-api-access-4p2mc" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566574 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963fd145-6705-467a-bac7-25b3ff8add39" volumeName="kubernetes.io/configmap/963fd145-6705-467a-bac7-25b3ff8add39-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566593 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9d50249-2496-4219-827f-72c9043805fe" volumeName="kubernetes.io/projected/a9d50249-2496-4219-827f-72c9043805fe-kube-api-access-vnmsw" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566605 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566642 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fda3848b-9d0b-4832-8789-862920d484b8" volumeName="kubernetes.io/secret/fda3848b-9d0b-4832-8789-862920d484b8-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566654 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566724 5719 manager.go:319] Starting recovery of all containers Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566735 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f65619e-5f82-4313-8a35-572fbece3755" volumeName="kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566748 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963fd145-6705-467a-bac7-25b3ff8add39" volumeName="kubernetes.io/projected/963fd145-6705-467a-bac7-25b3ff8add39-kube-api-access-dpqq8" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566764 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0284bc6-6a9b-42ed-b443-1f94367c8ac4" volumeName="kubernetes.io/projected/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-kube-api-access-2qsl4" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566787 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d142f31b-3461-4f5a-93e1-088ce6511100" volumeName="kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566824 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c4027c-b9b6-4a0d-a6d0-66f5a76004bf" volumeName="kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566850 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566887 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d142f31b-3461-4f5a-93e1-088ce6511100" volumeName="kubernetes.io/projected/d142f31b-3461-4f5a-93e1-088ce6511100-kube-api-access-hg94p" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566906 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f65619e-5f82-4313-8a35-572fbece3755" volumeName="kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566918 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f65619e-5f82-4313-8a35-572fbece3755" volumeName="kubernetes.io/projected/5f65619e-5f82-4313-8a35-572fbece3755-kube-api-access-h7htb" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566933 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f65619e-5f82-4313-8a35-572fbece3755" volumeName="kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566959 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64a235e0-e510-479f-a6bd-92cc4910a1e5" volumeName="kubernetes.io/projected/64a235e0-e510-479f-a6bd-92cc4910a1e5-kube-api-access-qk75x" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566971 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75963a1c-a5af-4c12-8adc-f6c5b9aa2e39" volumeName="kubernetes.io/empty-dir/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-volume-directive-shadow" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566984 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0284bc6-6a9b-42ed-b443-1f94367c8ac4" volumeName="kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-tmp" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.566998 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5476b645-f064-41af-b1da-9f7644f281d0" volumeName="kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-binary-copy" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567009 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567021 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 00:28:38.567007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567032 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567043 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368083a-8041-45bb-8e12-93f10e05eb15" volumeName="kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567053 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b42b8ff9-0140-4b1c-97f2-131bc141352f" volumeName="kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-env-overrides" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567066 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0284bc6-6a9b-42ed-b443-1f94367c8ac4" volumeName="kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-tuned" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567078 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567092 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204028bd-7b66-4354-bd39-54aaca835224" volumeName="kubernetes.io/projected/204028bd-7b66-4354-bd39-54aaca835224-kube-api-access-dz6mf" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567108 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aad04f16-056a-4115-97a0-e75d9e8b36b7" volumeName="kubernetes.io/configmap/aad04f16-056a-4115-97a0-e75d9e8b36b7-serviceca" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567119 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d142f31b-3461-4f5a-93e1-088ce6511100" volumeName="kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-tls" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567132 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fda3848b-9d0b-4832-8789-862920d484b8" volumeName="kubernetes.io/projected/fda3848b-9d0b-4832-8789-862920d484b8-kube-api-access-vdd6j" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567221 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567277 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1822938-c344-40c9-99c1-986663748169" volumeName="kubernetes.io/projected/b1822938-c344-40c9-99c1-986663748169-kube-api-access-rqx6g" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567318 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b75cde1b-aaee-46c8-b956-dcecf3b6afbf" volumeName="kubernetes.io/configmap/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-konnectivity-ca" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567357 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6d7d3bf-205d-4251-9b47-400892684e0e" volumeName="kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-cni-binary-copy" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567383 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567422 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368083a-8041-45bb-8e12-93f10e05eb15" volumeName="kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567447 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368083a-8041-45bb-8e12-93f10e05eb15" volumeName="kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567459 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="204028bd-7b66-4354-bd39-54aaca835224" volumeName="kubernetes.io/secret/204028bd-7b66-4354-bd39-54aaca835224-cert" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567472 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d142f31b-3461-4f5a-93e1-088ce6511100" volumeName="kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567498 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6d7d3bf-205d-4251-9b47-400892684e0e" volumeName="kubernetes.io/projected/d6d7d3bf-205d-4251-9b47-400892684e0e-kube-api-access-6krdc" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567544 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c4027c-b9b6-4a0d-a6d0-66f5a76004bf" volumeName="kubernetes.io/configmap/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-metrics-client-ca" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567557 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16395bf6-131d-4558-8bfe-83d994039370" volumeName="kubernetes.io/configmap/16395bf6-131d-4558-8bfe-83d994039370-config-volume" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567593 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f65619e-5f82-4313-8a35-572fbece3755" volumeName="kubernetes.io/empty-dir/5f65619e-5f82-4313-8a35-572fbece3755-audit-log" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567608 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567618 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b42b8ff9-0140-4b1c-97f2-131bc141352f" volumeName="kubernetes.io/projected/b42b8ff9-0140-4b1c-97f2-131bc141352f-kube-api-access-ngbqj" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567629 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567654 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567708 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567734 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6d7d3bf-205d-4251-9b47-400892684e0e" volumeName="kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-daemon-config" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567746 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d861409-3dd5-4eed-ada7-e1817b7fb58f" volumeName="kubernetes.io/projected/0d861409-3dd5-4eed-ada7-e1817b7fb58f-kube-api-access-trktv" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567756 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567767 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567790 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567801 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16395bf6-131d-4558-8bfe-83d994039370" volumeName="kubernetes.io/empty-dir/16395bf6-131d-4558-8bfe-83d994039370-tmp-dir" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567825 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5476b645-f064-41af-b1da-9f7644f281d0" volumeName="kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567850 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c4027c-b9b6-4a0d-a6d0-66f5a76004bf" volumeName="kubernetes.io/projected/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-kube-api-access-tcd45" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567861 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11796f3e-c26f-4453-81fc-b50c5ebfa01e" volumeName="kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567872 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1368083a-8041-45bb-8e12-93f10e05eb15" volumeName="kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567933 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567946 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d142f31b-3461-4f5a-93e1-088ce6511100" volumeName="kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-metrics-client-ca" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.567975 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0910c96-18d0-49b2-861d-0ec2a5793047" volumeName="kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.568020 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10c4027c-b9b6-4a0d-a6d0-66f5a76004bf" volumeName="kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.568035 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f65619e-5f82-4313-8a35-572fbece3755" volumeName="kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.568063 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.568117 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.568149 5719 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c01729b-d9cb-44ff-bcfd-3db050a479b8" volumeName="kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume" seLinuxMountContext="" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.568160 5719 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:28:38.569237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.568168 5719 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:28:38.592860 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.592808 5719 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:28:38.594229 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.594207 5719 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:28:38.594229 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.594224 5719 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:28:38.594345 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.594241 5719 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:28:38.594345 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.594274 5719 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:28:38.606315 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.606289 5719 generic.go:356] "Generic (PLEG): container finished" podID="ada14adc9fde4e31f8329dced7d0e766" containerID="cdd5cbd178dbeeb2ec0ef1150f777543f29d39fde2416ddf0a65669eebfbed3f" exitCode=0 Oct 31 00:28:38.638795 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.638765 5719 generic.go:356] "Generic (PLEG): container finished" podID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerID="122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d" exitCode=0 Oct 31 00:28:38.643774 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.641768 5719 generic.go:356] "Generic (PLEG): container finished" podID="d142f31b-3461-4f5a-93e1-088ce6511100" containerID="f9fae05c0daf26caca56cc9e2078979df05366472eb9053101ee2d12718467cd" exitCode=0 Oct 31 00:28:38.658056 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.658001 5719 generic.go:356] "Generic (PLEG): container finished" podID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerID="6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9" exitCode=0 Oct 31 00:28:38.662532 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.662511 5719 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-8.us-west-2.compute.internal\" not found" Oct 31 00:28:38.663898 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.663852 5719 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="7856a5b9e384a2f5cc944ccba6518e1b4323906c20fcb8d5e83aa6ed2085fb91" exitCode=0 Oct 31 00:28:38.663898 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.663881 5719 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="a9d5ec98d3e248f651fcde3acd51ffda93adfa6ca99d28ccd14268237f4d9d75" exitCode=0 Oct 31 00:28:38.663898 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.663893 5719 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="042b672e24a45f9b98a0f0c55a101333dd651a25ad2d9584c05be1d7f82a0b59" exitCode=0 Oct 31 00:28:38.663898 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.663902 5719 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="a9c4bc84c620042da09dfb88c7c12aa88d0f50aee414ef1aac74c619dc710f4d" exitCode=0 Oct 31 00:28:38.664219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.663911 5719 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="1d5274c8a659aaf6fbcbd518031517f8545c507beabf9dc576c7565fcc7d5266" exitCode=0 Oct 31 00:28:38.664219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.663920 5719 generic.go:356] "Generic (PLEG): container finished" podID="5476b645-f064-41af-b1da-9f7644f281d0" containerID="b26ca43ba5ad9171c768c288bd6a8ab593eb06e793388bc4af524069198b7d57" exitCode=0 Oct 31 00:28:38.684393 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.684373 5719 manager.go:324] Recovery completed Oct 31 00:28:38.694590 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.694562 5719 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 00:28:38.703723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.703707 5719 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:38.705576 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.705559 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:38.705635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.705590 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:38.705635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.705600 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:38.708226 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708210 5719 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:28:38.708226 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708224 5719 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:28:38.708378 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708226 5719 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:38.708378 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708243 5719 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:28:38.708481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708389 5719 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:28:38.708481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708404 5719 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:28:38.708481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708427 5719 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:28:38.708481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708436 5719 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:28:38.708481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708444 5719 policy_none.go:50] "Start" Oct 31 00:28:38.708481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708457 5719 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:28:38.708481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708470 5719 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:28:38.708813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708644 5719 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:28:38.708813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708655 5719 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:28:38.708813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.708664 5719 policy_none.go:44] "Start" Oct 31 00:28:38.709231 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.709214 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:38.709284 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.709241 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:38.709284 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.709251 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:38.714936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.714910 5719 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:28:38.718823 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.718809 5719 manager.go:341] "Starting Device Plugin manager" Oct 31 00:28:38.718935 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.718837 5719 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:28:38.718935 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.718849 5719 server.go:87] "Starting device plugin registration server" Oct 31 00:28:38.719139 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.719125 5719 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:28:38.719201 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.719139 5719 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:28:38.719268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.719234 5719 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:28:38.719512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.719330 5719 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:28:38.719626 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.719524 5719 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:28:38.719626 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.719529 5719 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:28:38.720105 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.719567 5719 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:28:38.719507631Z","UUID":"25cedf3d-5df4-4e2a-89d5-461c1511d3a3","Handler":null,"Name":"","Endpoint":""} Oct 31 00:28:38.720956 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.720937 5719 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:28:38.721072 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.720979 5719 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-8.us-west-2.compute.internal\" not found" Oct 31 00:28:38.721368 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.721344 5719 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:28:38.721427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.721378 5719 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:28:38.732083 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.732063 5719 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:38.733040 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.733023 5719 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:38.780559 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.780537 5719 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:38.780638 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.780583 5719 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:28:38.819264 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.819247 5719 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:28:38.820149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.820132 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:28:38.820209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.820159 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:28:38.820209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.820170 5719 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-8.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 31 00:28:38.820265 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.820212 5719 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.826456 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.826441 5719 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:28:38.894947 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.894907 5719 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal"] Oct 31 00:28:38.895512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.895481 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"ada14adc9fde4e31f8329dced7d0e766","Type":"ContainerStarted","Data":"35dbcec67c9c79ac3646c512b5d2065d67729b71a3275fa89d806e30bf13359d"} Oct 31 00:28:38.895548 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.895525 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"ada14adc9fde4e31f8329dced7d0e766","Type":"ContainerDied","Data":"cdd5cbd178dbeeb2ec0ef1150f777543f29d39fde2416ddf0a65669eebfbed3f"} Oct 31 00:28:38.895577 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.895547 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"ada14adc9fde4e31f8329dced7d0e766","Type":"ContainerStarted","Data":"4ddd7ba8d34be7c0899e8c77bb5d0bd116657b2b0bf891271a6147ae1fbe39a8"} Oct 31 00:28:38.895577 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.895560 5719 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.895651 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.895639 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"eaf6c79eda58382b69cd6ec1e1ffc85d","Type":"ContainerStarted","Data":"4f0ebf687d99c395d11016ed38d9938dea8ac8749902332c9853e36138740936"} Oct 31 00:28:38.895720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.895654 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" event={"ID":"eaf6c79eda58382b69cd6ec1e1ffc85d","Type":"ContainerStarted","Data":"e059d0603b0e71ebf91d721d4911a8d57b734f52c1589e2e179a8aef6b367802"} Oct 31 00:28:38.895818 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.895803 5719 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.936541 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.936491 5719 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.936605 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.936547 5719 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.966389 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.966371 5719 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:38.966474 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.966408 5719 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.967235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.967225 5719 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:38.967267 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:38.967249 5719 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.981397 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.981377 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.981441 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.981403 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:38.981441 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:38.981422 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaf6c79eda58382b69cd6ec1e1ffc85d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"eaf6c79eda58382b69cd6ec1e1ffc85d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.082228 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.082198 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.082228 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.082228 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaf6c79eda58382b69cd6ec1e1ffc85d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"eaf6c79eda58382b69cd6ec1e1ffc85d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.082427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.082244 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.082427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.082282 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.082427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.082318 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaf6c79eda58382b69cd6ec1e1ffc85d-config\") pod \"kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"eaf6c79eda58382b69cd6ec1e1ffc85d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.082427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.082318 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ada14adc9fde4e31f8329dced7d0e766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" (UID: \"ada14adc9fde4e31f8329dced7d0e766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.529602 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.529572 5719 apiserver.go:52] "Watching apiserver" Oct 31 00:28:39.668403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.668370 5719 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.669269 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.669251 5719 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.677996 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.677978 5719 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:28:39.678600 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.678581 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-68qwm","openshift-monitoring/metrics-server-d46547967-pkq4f","openshift-monitoring/node-exporter-mqb6s","openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt","openshift-monitoring/alertmanager-main-0","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4","openshift-insights/insights-runtime-extractor-47ws4","openshift-multus/multus-8slwz","openshift-multus/network-metrics-daemon-ccs74","openshift-dns/dns-default-5jqlp","kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-k4z7d","openshift-ovn-kubernetes/ovnkube-node-vk9bb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal","openshift-console/console-697bb4b4df-4gpsf","openshift-image-registry/node-ca-7x6x2","openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh","openshift-network-diagnostics/network-check-target-gzjf9","kube-system/konnectivity-agent-m98mn","openshift-cluster-node-tuning-operator/tuned-9dhhl","kube-system/global-pull-secret-syncer-ht56p","openshift-dns/node-resolver-m7t4w","openshift-multus/multus-additional-cni-plugins-tdb56","openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv"] Oct 31 00:28:39.728110 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.728090 5719 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:39.728256 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.728146 5719 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.731599 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.731585 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dwmlr\"" Oct 31 00:28:39.731655 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.731611 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:28:39.732446 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.732427 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cdw8b\"" Oct 31 00:28:39.732485 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.732458 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rlgrr\"" Oct 31 00:28:39.733372 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.733355 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-46b97\"" Oct 31 00:28:39.733503 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.733484 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:28:39.733680 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.733664 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m2wdr\"" Oct 31 00:28:39.733737 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.733709 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jsqnl\"" Oct 31 00:28:39.733769 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.733713 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:28:39.733814 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.733782 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-78fsr\"" Oct 31 00:28:39.733909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.733897 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:28:39.733975 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.733964 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lwt4s\"" Oct 31 00:28:39.734437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734419 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.734496 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734446 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.734545 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734529 5719 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:28:39.734597 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.734575 5719 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-8.us-west-2.compute.internal" Oct 31 00:28:39.734715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734534 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.734773 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734754 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:28:39.734823 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734760 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9qgw5\"" Oct 31 00:28:39.734880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734821 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:28:39.734880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734827 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.734880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.734830 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.735148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.735128 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.735248 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.735178 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:28:39.735741 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.735720 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:28:39.735920 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.735902 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:28:39.736035 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736009 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:28:39.736252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736229 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.736408 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736290 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.736408 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736325 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.736408 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736356 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.736566 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736533 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2znq\"" Oct 31 00:28:39.736618 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736562 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.736618 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736596 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.736744 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736623 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6gdh\"" Oct 31 00:28:39.736853 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.736826 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:28:39.737040 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737015 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rw8bx\"" Oct 31 00:28:39.737110 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737101 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.737222 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737206 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:28:39.737275 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737250 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:28:39.737275 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737260 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.737370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737205 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.737526 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737509 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.737611 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737598 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:28:39.737676 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737638 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nwzth\"" Oct 31 00:28:39.738015 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.737999 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:28:39.765274 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.765255 5719 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:28:39.767312 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.767298 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:28:39.767978 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.767965 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:28:39.768022 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.767980 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:28:39.768487 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.768464 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:28:39.768567 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.768525 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hg2s2\"" Oct 31 00:28:39.768905 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.768891 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-75n8c\"" Oct 31 00:28:39.768949 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.768923 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52f7q\"" Oct 31 00:28:39.770019 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.770005 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:28:39.770621 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.770609 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:28:39.770954 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.770942 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:28:39.771038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.771026 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:28:39.771072 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.771038 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:28:39.771109 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.771073 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5h47b\"" Oct 31 00:28:39.775151 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.775138 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:28:39.776104 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.776092 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:28:39.776704 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.776675 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7hhvp\"" Oct 31 00:28:39.777473 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.777461 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:28:39.778106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.778095 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.778281 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.778266 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.779898 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.779852 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.779959 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.779905 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.780015 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.779997 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:28:39.780061 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.780039 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:28:39.780122 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.780064 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:28:39.780122 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.780065 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:28:39.780195 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.780173 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:28:39.780223 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.780213 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:28:39.780259 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.780246 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:28:39.782923 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.782897 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:28:39.783013 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.782956 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:28:39.783013 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.782996 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:28:39.783176 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.783164 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:28:39.783369 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.783358 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:28:39.785546 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785526 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-socket-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.785630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785561 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.785630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785609 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.785766 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785638 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-ovn\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.785766 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785665 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.785766 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785713 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-host\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.785766 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785743 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-hostroot\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785784 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785786 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-conf-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785813 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785827 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-daemon-config\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785856 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785883 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785912 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785940 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-systemd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.785966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785963 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-node-log\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785997 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-netns\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.785992 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786022 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-daemon-config\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786027 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395bf6-131d-4558-8bfe-83d994039370-metrics-tls\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786048 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5k99\" (UniqueName: \"kubernetes.io/projected/d0910c96-18d0-49b2-861d-0ec2a5793047-kube-api-access-q5k99\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786065 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786080 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-sys-fs\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786096 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786147 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786208 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysconfig\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786224 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-tuned\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.786235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786230 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16395bf6-131d-4558-8bfe-83d994039370-metrics-tls\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786273 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9d50249-2496-4219-827f-72c9043805fe-data-volume\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786293 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsl4\" (UniqueName: \"kubernetes.io/projected/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-kube-api-access-2qsl4\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786317 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6vv\" (UniqueName: \"kubernetes.io/projected/aad04f16-056a-4115-97a0-e75d9e8b36b7-kube-api-access-5z6vv\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786343 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-tuned\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786373 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786398 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9d50249-2496-4219-827f-72c9043805fe-data-volume\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786445 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786494 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-device-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786524 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-system-cni-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786550 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786585 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-env-overrides\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786637 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-bin\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786638 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786662 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqx6g\" (UniqueName: \"kubernetes.io/projected/b1822938-c344-40c9-99c1-986663748169-kube-api-access-rqx6g\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786706 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786736 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786746 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786760 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-kubernetes\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786785 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-env-overrides\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.786784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786795 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-bin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786890 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786895 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786930 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0910c96-18d0-49b2-861d-0ec2a5793047-tmp\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786952 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-var-lib-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786961 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786977 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.786994 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787005 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0910c96-18d0-49b2-861d-0ec2a5793047-tmp\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787011 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-tls\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787034 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-konnectivity-ca\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787057 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1822938-c344-40c9-99c1-986663748169-hosts-file\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787085 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-registration-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787109 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-kubelet\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787203 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787208 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4gq2\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787241 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f65619e-5f82-4313-8a35-572fbece3755-audit-log\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787259 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-dbus\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787282 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787332 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787339 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-konnectivity-ca\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787339 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5f65619e-5f82-4313-8a35-572fbece3755-audit-log\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787359 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-sys\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787357 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787384 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-kubelet\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787403 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-tls\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.787495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787456 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787552 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787599 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.787619 5719 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787622 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-tls\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787643 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.787711 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.287662584 +0000 UTC m=+1.823804900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787751 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787766 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787771 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbng\" (UniqueName: \"kubernetes.io/projected/5476b645-f064-41af-b1da-9f7644f281d0-kube-api-access-chbng\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787789 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-systemd-units\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.787802 5719 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787815 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngbqj\" (UniqueName: \"kubernetes.io/projected/b42b8ff9-0140-4b1c-97f2-131bc141352f-kube-api-access-ngbqj\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.787851 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.287840538 +0000 UTC m=+1.823982822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"service-ca" not registered Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787864 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-var-lib-kubelet\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787882 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-wtmp\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787909 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787938 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787956 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad04f16-056a-4115-97a0-e75d9e8b36b7-host\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.787971 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-netd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788067 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9d50249-2496-4219-827f-72c9043805fe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788087 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788103 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788120 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd6j\" (UniqueName: \"kubernetes.io/projected/fda3848b-9d0b-4832-8789-862920d484b8-kube-api-access-vdd6j\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788140 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788143 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788167 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/963fd145-6705-467a-bac7-25b3ff8add39-iptables-alerter-script\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788209 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788239 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-k8s-cni-cncf-io\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788266 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-textfile\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788297 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-accelerators-collector-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788305 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a9d50249-2496-4219-827f-72c9043805fe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788342 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788346 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-original-pull-secret\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788354 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2mc\" (UniqueName: \"kubernetes.io/projected/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-api-access-4p2mc\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788392 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788408 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-textfile\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788458 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788477 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-accelerators-collector-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.788510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788490 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-netns\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788599 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5476b645-f064-41af-b1da-9f7644f281d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788616 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqq8\" (UniqueName: \"kubernetes.io/projected/963fd145-6705-467a-bac7-25b3ff8add39-kube-api-access-dpqq8\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788646 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788673 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-system-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788684 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/963fd145-6705-467a-bac7-25b3ff8add39-iptables-alerter-script\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788715 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788753 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788815 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788817 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-client-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788848 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7htb\" (UniqueName: \"kubernetes.io/projected/5f65619e-5f82-4313-8a35-572fbece3755-kube-api-access-h7htb\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788867 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1822938-c344-40c9-99c1-986663748169-tmp-dir\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788888 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788905 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-etc-kubernetes\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788920 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-metrics-client-ca\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788944 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fda3848b-9d0b-4832-8789-862920d484b8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788976 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.788982 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789016 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmsw\" (UniqueName: \"kubernetes.io/projected/a9d50249-2496-4219-827f-72c9043805fe-kube-api-access-vnmsw\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789024 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789050 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1822938-c344-40c9-99c1-986663748169-tmp-dir\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789068 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-cnibin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789102 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95dhq\" (UniqueName: \"kubernetes.io/projected/16395bf6-131d-4558-8bfe-83d994039370-kube-api-access-95dhq\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789129 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789142 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d142f31b-3461-4f5a-93e1-088ce6511100-metrics-client-ca\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789145 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789169 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk75x\" (UniqueName: \"kubernetes.io/projected/64a235e0-e510-479f-a6bd-92cc4910a1e5-kube-api-access-qk75x\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789218 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789301 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-script-lib\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789326 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trktv\" (UniqueName: \"kubernetes.io/projected/0d861409-3dd5-4eed-ada7-e1817b7fb58f-kube-api-access-trktv\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789342 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789370 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-multus\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789406 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-root\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789431 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-client-certs\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789507 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-script-lib\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789543 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789588 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789593 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9d50249-2496-4219-827f-72c9043805fe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789637 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-cni-binary-copy\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789662 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krdc\" (UniqueName: \"kubernetes.io/projected/d6d7d3bf-205d-4251-9b47-400892684e0e-kube-api-access-6krdc\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789703 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-run\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789729 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-slash\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789754 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789778 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789825 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6d7d3bf-205d-4251-9b47-400892684e0e-cni-binary-copy\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789846 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zs9\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789851 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a9d50249-2496-4219-827f-72c9043805fe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789884 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.789983 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-log-socket\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790027 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.790041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790053 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790129 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-multus-certs\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790160 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790226 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790260 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/963fd145-6705-467a-bac7-25b3ff8add39-host-slash\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790285 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9d50249-2496-4219-827f-72c9043805fe-crio-socket\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790328 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790357 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aad04f16-056a-4115-97a0-e75d9e8b36b7-serviceca\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790380 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790419 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790430 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790501 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790519 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-conf\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790534 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-systemd\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790579 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-agent-certs\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790597 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790613 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-etc-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790631 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790646 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.790660 5719 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790680 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-tmp\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.790734 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.290719332 +0000 UTC m=+1.826861636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790756 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-socket-dir-parent\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790758 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-tmp\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790790 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0910c96-18d0-49b2-861d-0ec2a5793047-metrics-client-ca\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790824 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790832 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b75cde1b-aaee-46c8-b956-dcecf3b6afbf-agent-certs\") pod \"konnectivity-agent-m98mn\" (UID: \"b75cde1b-aaee-46c8-b956-dcecf3b6afbf\") " pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790853 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovn-node-metrics-cert\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790885 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6mf\" (UniqueName: \"kubernetes.io/projected/204028bd-7b66-4354-bd39-54aaca835224-kube-api-access-dz6mf\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790892 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d861409-3dd5-4eed-ada7-e1817b7fb58f-metrics-certs\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.790908 5719 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790911 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790939 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.790954 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.290939876 +0000 UTC m=+1.827082184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790966 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0910c96-18d0-49b2-861d-0ec2a5793047-metrics-client-ca\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790980 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-modprobe-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.790992 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791008 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791016 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/aad04f16-056a-4115-97a0-e75d9e8b36b7-serviceca\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791037 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791053 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791069 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791094 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791121 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovn-node-metrics-cert\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791120 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791182 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791182 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-grpc-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791212 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-cnibin\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791256 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-sys\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791273 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791292 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791326 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg94p\" (UniqueName: \"kubernetes.io/projected/d142f31b-3461-4f5a-93e1-088ce6511100-kube-api-access-hg94p\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791354 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791418 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791444 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-metrics-server-audit-profiles\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791464 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-kubelet-config\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791481 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791485 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791496 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791520 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/204028bd-7b66-4354-bd39-54aaca835224-cert\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791542 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395bf6-131d-4558-8bfe-83d994039370-config-volume\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.791557 5719 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791560 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcd45\" (UniqueName: \"kubernetes.io/projected/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-kube-api-access-tcd45\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.791595 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.291585915 +0000 UTC m=+1.827728197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791669 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791718 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/204028bd-7b66-4354-bd39-54aaca835224-cert\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791725 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-config\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791736 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16395bf6-131d-4558-8bfe-83d994039370-config-volume\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791763 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-lib-modules\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.791764 5719 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791793 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5f65619e-5f82-4313-8a35-572fbece3755-metrics-server-audit-profiles\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791802 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395bf6-131d-4558-8bfe-83d994039370-tmp-dir\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.791837 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.291828568 +0000 UTC m=+1.827970851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-config" not registered Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791860 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791863 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b42b8ff9-0140-4b1c-97f2-131bc141352f-ovnkube-config\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791887 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16395bf6-131d-4558-8bfe-83d994039370-tmp-dir\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:39.792049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.791996 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:39.818649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.818620 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:28:39.818763 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.818686 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:28:39.819681 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.819660 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:28:39.819842 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.819826 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.821076 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.821056 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.823945 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.823929 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.833453 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.833430 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:28:39.833884 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.833870 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:28:39.833953 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.833925 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:28:39.834041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834026 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:28:39.834127 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834113 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:28:39.834173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834115 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:28:39.834316 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834302 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:28:39.834381 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834368 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:28:39.834433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834396 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:28:39.834433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834410 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:28:39.834433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834418 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qnk5n\"" Oct 31 00:28:39.834433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834419 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:28:39.834572 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834464 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:28:39.834572 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.834476 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:28:39.835839 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.835827 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:28:39.836108 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.836093 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.836108 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.836103 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x4kkw\"" Oct 31 00:28:39.836174 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.836136 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:28:39.836202 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.836178 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.836301 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.836289 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.837032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.837017 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kqch6\"" Oct 31 00:28:39.837113 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.837101 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.837159 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.837123 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.837217 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.837204 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.837521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.837507 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:28:39.838321 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.838307 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.839432 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.839419 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.840010 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.839988 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:28:39.840062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.839992 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:28:39.840147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.840132 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.840177 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.840140 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-client-certs\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.840250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.840238 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.840326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.840314 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.840603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.840590 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.840674 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.840663 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.841538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.841522 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:28:39.841589 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.841525 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9q7lj\"" Oct 31 00:28:39.841589 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.841572 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:28:39.841661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.841587 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.841661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.841630 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.842299 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.842288 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-djuj1hhvgpfmh\"" Oct 31 00:28:39.842329 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.842290 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dqugrurpbd40l\"" Oct 31 00:28:39.843050 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.843038 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:28:39.844644 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.844630 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:28:39.844682 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.844643 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3ank9btekulg4\"" Oct 31 00:28:39.845748 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.845733 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5k99\" (UniqueName: \"kubernetes.io/projected/d0910c96-18d0-49b2-861d-0ec2a5793047-kube-api-access-q5k99\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.845748 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.845743 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqx6g\" (UniqueName: \"kubernetes.io/projected/b1822938-c344-40c9-99c1-986663748169-kube-api-access-rqx6g\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:28:39.845903 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.845890 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.846604 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.846591 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsl4\" (UniqueName: \"kubernetes.io/projected/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-kube-api-access-2qsl4\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.847332 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.847318 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-secret-metrics-server-tls\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.847578 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.847564 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbng\" (UniqueName: \"kubernetes.io/projected/5476b645-f064-41af-b1da-9f7644f281d0-kube-api-access-chbng\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.847743 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.847727 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.848272 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.848258 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6vv\" (UniqueName: \"kubernetes.io/projected/aad04f16-056a-4115-97a0-e75d9e8b36b7-kube-api-access-5z6vv\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:28:39.849117 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.849104 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5f65619e-5f82-4313-8a35-572fbece3755-client-ca-bundle\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.850513 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.849533 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fda3848b-9d0b-4832-8789-862920d484b8-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:39.850513 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.850464 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqq8\" (UniqueName: \"kubernetes.io/projected/963fd145-6705-467a-bac7-25b3ff8add39-kube-api-access-dpqq8\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:28:39.850638 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.850524 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87bx\" (UniqueName: \"kubernetes.io/projected/8b9dc277-9345-44d1-9f7e-90b5a9f3380e-kube-api-access-f87bx\") pod \"network-check-target-gzjf9\" (UID: \"8b9dc277-9345-44d1-9f7e-90b5a9f3380e\") " pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:39.850734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.850718 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.850784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.850733 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.850838 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.850818 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-thanos-querier-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.853715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.851178 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2mc\" (UniqueName: \"kubernetes.io/projected/75963a1c-a5af-4c12-8adc-f6c5b9aa2e39-kube-api-access-4p2mc\") pod \"kube-state-metrics-5dc8d7d89b-9klj5\" (UID: \"75963a1c-a5af-4c12-8adc-f6c5b9aa2e39\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9klj5" Oct 31 00:28:39.853715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.851633 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd6j\" (UniqueName: \"kubernetes.io/projected/fda3848b-9d0b-4832-8789-862920d484b8-kube-api-access-vdd6j\") pod \"monitoring-plugin-77f4f76594-hxlqh\" (UID: \"fda3848b-9d0b-4832-8789-862920d484b8\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:39.853715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.851874 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0910c96-18d0-49b2-861d-0ec2a5793047-secret-grpc-tls\") pod \"thanos-querier-6dcb67b5b7-gwgv4\" (UID: \"d0910c96-18d0-49b2-861d-0ec2a5793047\") " pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:39.853715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.852605 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7htb\" (UniqueName: \"kubernetes.io/projected/5f65619e-5f82-4313-8a35-572fbece3755-kube-api-access-h7htb\") pod \"metrics-server-d46547967-pkq4f\" (UID: \"5f65619e-5f82-4313-8a35-572fbece3755\") " pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:39.853715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.853051 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngbqj\" (UniqueName: \"kubernetes.io/projected/b42b8ff9-0140-4b1c-97f2-131bc141352f-kube-api-access-ngbqj\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.853715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.853372 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trktv\" (UniqueName: \"kubernetes.io/projected/0d861409-3dd5-4eed-ada7-e1817b7fb58f-kube-api-access-trktv\") pod \"network-metrics-daemon-ccs74\" (UID: \"0d861409-3dd5-4eed-ada7-e1817b7fb58f\") " pod="openshift-multus/network-metrics-daemon-ccs74" Oct 31 00:28:39.854167 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.854151 5719 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:28:39.854225 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.854169 5719 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:28:39.854225 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.854180 5719 projected.go:196] Error preparing data for projected volume kube-api-access-klpkz for pod openshift-console/console-697bb4b4df-4gpsf: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:39.854307 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:39.854232 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:40.354221437 +0000 UTC m=+1.890363763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-klpkz" (UniqueName: "kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:39.854653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.854630 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95dhq\" (UniqueName: \"kubernetes.io/projected/16395bf6-131d-4558-8bfe-83d994039370-kube-api-access-95dhq\") pod \"dns-default-5jqlp\" (UID: \"16395bf6-131d-4558-8bfe-83d994039370\") " pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:39.854797 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.854782 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zs9\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9\") pod \"alertmanager-main-0\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:28:39.855514 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.855478 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krdc\" (UniqueName: \"kubernetes.io/projected/d6d7d3bf-205d-4251-9b47-400892684e0e-kube-api-access-6krdc\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.855617 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.855599 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg94p\" (UniqueName: \"kubernetes.io/projected/d142f31b-3461-4f5a-93e1-088ce6511100-kube-api-access-hg94p\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.855786 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.855770 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcd45\" (UniqueName: \"kubernetes.io/projected/10c4027c-b9b6-4a0d-a6d0-66f5a76004bf-kube-api-access-tcd45\") pod \"openshift-state-metrics-7bd7489957-lspcv\" (UID: \"10c4027c-b9b6-4a0d-a6d0-66f5a76004bf\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lspcv" Oct 31 00:28:39.855891 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.855873 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6mf\" (UniqueName: \"kubernetes.io/projected/204028bd-7b66-4354-bd39-54aaca835224-kube-api-access-dz6mf\") pod \"ingress-canary-k4z7d\" (UID: \"204028bd-7b66-4354-bd39-54aaca835224\") " pod="openshift-ingress-canary/ingress-canary-k4z7d" Oct 31 00:28:39.859323 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.859305 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tjsbm\"" Oct 31 00:28:39.859408 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.859376 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk75x\" (UniqueName: \"kubernetes.io/projected/64a235e0-e510-479f-a6bd-92cc4910a1e5-kube-api-access-qk75x\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.862346 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.862327 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnmsw\" (UniqueName: \"kubernetes.io/projected/a9d50249-2496-4219-827f-72c9043805fe-kube-api-access-vnmsw\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.862536 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.862521 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4gq2\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2\") pod \"prometheus-k8s-0\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:39.893174 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893152 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-conf\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893175 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-systemd\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893222 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-etc-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.893357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893255 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-etc-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.893357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893273 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-systemd\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893308 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-socket-dir-parent\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.893357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893340 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-modprobe-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893348 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-conf\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893356 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893397 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-cnibin\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893401 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-socket-dir-parent\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893431 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-sys\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893440 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-cnibin\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893458 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-kubelet-config\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893469 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysctl-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893489 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-kubelet-config\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893490 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-sys\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893461 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-modprobe-d\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893493 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893528 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893537 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-lib-modules\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893563 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-socket-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.893597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893588 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-ovn\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893610 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-host\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893619 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-lib-modules\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893632 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-hostroot\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893636 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-ovn\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893647 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-host\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893648 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-socket-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893655 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-conf-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893664 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-hostroot\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893714 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-systemd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893719 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-conf-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893730 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-node-log\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893735 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-systemd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893745 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-netns\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893757 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-node-log\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893763 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893785 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-sys-fs\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893797 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-netns\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893808 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysconfig\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893835 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-device-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893852 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-sys-fs\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893854 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893861 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-system-cni-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893879 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-sysconfig\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893891 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893891 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-system-cni-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893911 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-device-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893928 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893933 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-bin\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893954 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-kubernetes\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.893955 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-bin\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894003 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-etc-kubernetes\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894020 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-bin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894037 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-var-lib-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894063 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-bin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894064 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-var-lib-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894074 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1822938-c344-40c9-99c1-986663748169-hosts-file\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894097 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1822938-c344-40c9-99c1-986663748169-hosts-file\") pod \"node-resolver-m7t4w\" (UID: \"b1822938-c344-40c9-99c1-986663748169\") " pod="openshift-dns/node-resolver-m7t4w" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894098 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-registration-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894116 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-kubelet\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894127 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64a235e0-e510-479f-a6bd-92cc4910a1e5-registration-dir\") pod \"aws-ebs-csi-driver-node-4fbrt\" (UID: \"64a235e0-e510-479f-a6bd-92cc4910a1e5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fbrt" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894132 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-dbus\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894146 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-sys\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894150 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-kubelet\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.894147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894160 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-kubelet\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894182 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894190 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5-dbus\") pod \"global-pull-secret-syncer-ht56p\" (UID: \"8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5\") " pod="kube-system/global-pull-secret-syncer-ht56p" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894208 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-sys\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894217 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-systemd-units\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894239 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-var-lib-kubelet\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894221 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5476b645-f064-41af-b1da-9f7644f281d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tdb56\" (UID: \"5476b645-f064-41af-b1da-9f7644f281d0\") " pod="openshift-multus/multus-additional-cni-plugins-tdb56" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894208 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-kubelet\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894286 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-wtmp\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894254 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-node-exporter-wtmp\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894298 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-systemd-units\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894308 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad04f16-056a-4115-97a0-e75d9e8b36b7-host\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894312 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-var-lib-kubelet\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894324 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-netd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894340 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894356 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-k8s-cni-cncf-io\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894390 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-cni-netd\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894395 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-multus-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894406 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aad04f16-056a-4115-97a0-e75d9e8b36b7-host\") pod \"node-ca-7x6x2\" (UID: \"aad04f16-056a-4115-97a0-e75d9e8b36b7\") " pod="openshift-image-registry/node-ca-7x6x2" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894406 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-k8s-cni-cncf-io\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894429 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-netns\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894448 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-system-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894464 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-etc-kubernetes\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894476 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-run-netns\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894486 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-cnibin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894500 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-etc-kubernetes\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894506 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894505 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-system-cni-dir\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894523 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-multus\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894528 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-cnibin\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894545 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-root\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894558 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894568 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-run\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894583 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-slash\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894603 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894614 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d142f31b-3461-4f5a-93e1-088ce6511100-root\") pod \"node-exporter-mqb6s\" (UID: \"d142f31b-3461-4f5a-93e1-088ce6511100\") " pod="openshift-monitoring/node-exporter-mqb6s" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894637 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-var-lib-cni-multus\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894646 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0284bc6-6a9b-42ed-b443-1f94367c8ac4-run\") pod \"tuned-9dhhl\" (UID: \"d0284bc6-6a9b-42ed-b443-1f94367c8ac4\") " pod="openshift-cluster-node-tuning-operator/tuned-9dhhl" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894652 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-run-openvswitch\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894655 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-host-slash\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894646 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-log-socket\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894686 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b42b8ff9-0140-4b1c-97f2-131bc141352f-log-socket\") pod \"ovnkube-node-vk9bb\" (UID: \"b42b8ff9-0140-4b1c-97f2-131bc141352f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894722 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-multus-certs\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894748 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d7d3bf-205d-4251-9b47-400892684e0e-host-run-multus-certs\") pod \"multus-8slwz\" (UID: \"d6d7d3bf-205d-4251-9b47-400892684e0e\") " pod="openshift-multus/multus-8slwz" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894750 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/963fd145-6705-467a-bac7-25b3ff8add39-host-slash\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894774 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9d50249-2496-4219-827f-72c9043805fe-crio-socket\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894794 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/963fd145-6705-467a-bac7-25b3ff8add39-host-slash\") pod \"iptables-alerter-68qwm\" (UID: \"963fd145-6705-467a-bac7-25b3ff8add39\") " pod="openshift-network-operator/iptables-alerter-68qwm" Oct 31 00:28:39.895515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.894851 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a9d50249-2496-4219-827f-72c9043805fe-crio-socket\") pod \"insights-runtime-extractor-47ws4\" (UID: \"a9d50249-2496-4219-827f-72c9043805fe\") " pod="openshift-insights/insights-runtime-extractor-47ws4" Oct 31 00:28:39.992965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:39.992951 5719 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:28:40.298244 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:40.298212 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:40.298244 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:40.298249 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:40.298462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:40.298324 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:40.298462 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298338 5719 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:40.298462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:40.298354 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:40.298462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:40.298389 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:40.298462 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298407 5719 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:28:40.298462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:40.298416 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:40.298462 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298430 5719 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:40.298462 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298464 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.298447316 +0000 UTC m=+2.834589602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"service-ca" not registered Oct 31 00:28:40.298869 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298486 5719 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:28:40.298869 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298488 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.298480177 +0000 UTC m=+2.834622460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:40.298869 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298498 5719 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:40.298869 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298547 5719 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:40.298869 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298505 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.298500393 +0000 UTC m=+2.834642676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:40.298869 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298614 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.298583482 +0000 UTC m=+2.834725770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-config" not registered Oct 31 00:28:40.298869 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298644 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.298632553 +0000 UTC m=+2.834774841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:40.298869 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.298663 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.298654524 +0000 UTC m=+2.834796807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:40.400015 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:40.399968 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:40.400145 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.400103 5719 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:28:40.400145 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.400122 5719 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:28:40.400145 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.400133 5719 projected.go:196] Error preparing data for projected volume kube-api-access-klpkz for pod openshift-console/console-697bb4b4df-4gpsf: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:40.400282 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:40.400187 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:41.400169223 +0000 UTC m=+2.936311529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-klpkz" (UniqueName: "kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:40.924208 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:40.924179 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:28:41.288653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.288592 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:41.309632 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.309605 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:41.309769 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.309650 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:41.309769 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.309756 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:41.309864 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.309787 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:41.309864 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.309759 5719 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:41.309937 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.309881 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:43.309862834 +0000 UTC m=+4.846005120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:41.309937 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.309765 5719 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:28:41.309937 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.309903 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:41.309937 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.309917 5719 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.309938 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.309955 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:43.309938903 +0000 UTC m=+4.846081195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-config" not registered Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.309790 5719 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.309985 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:43.309975529 +0000 UTC m=+4.846117817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"service-ca" not registered Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.309991 5719 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.310010 5719 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.310031 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:43.310017911 +0000 UTC m=+4.846160208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.310048 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:43.310036547 +0000 UTC m=+4.846178847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:41.310086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.310060 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:43.310053591 +0000 UTC m=+4.846195873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:41.411218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.411189 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:41.411364 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.411352 5719 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:28:41.411419 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.411372 5719 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:28:41.411419 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.411386 5719 projected.go:196] Error preparing data for projected volume kube-api-access-klpkz for pod openshift-console/console-697bb4b4df-4gpsf: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:41.411490 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:41.411436 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:43.411420848 +0000 UTC m=+4.947563150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-klpkz" (UniqueName: "kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:41.523876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:41.523848 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5jqlp" Oct 31 00:28:43.129712 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.129663 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:43.134865 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.134844 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6dcb67b5b7-gwgv4" Oct 31 00:28:43.331974 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.331948 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:43.332121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.332032 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:43.332121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.332055 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:43.332121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.332077 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:43.332121 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332087 5719 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:28:43.332121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.332096 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332140 5719 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332169 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.332148841 +0000 UTC m=+8.868291147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"service-ca" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332184 5719 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332193 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.332177723 +0000 UTC m=+8.868320022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.332211 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332229 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.332216795 +0000 UTC m=+8.868359096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332229 5719 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332240 5719 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332184 5719 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332266 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.332255909 +0000 UTC m=+8.868398192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332283 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.332274625 +0000 UTC m=+8.868416912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:43.332382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.332296 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.33228878 +0000 UTC m=+8.868431063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-config" not registered Oct 31 00:28:43.433401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:43.433333 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:43.433551 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.433496 5719 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:28:43.433551 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.433520 5719 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:28:43.433551 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.433532 5719 projected.go:196] Error preparing data for projected volume kube-api-access-klpkz for pod openshift-console/console-697bb4b4df-4gpsf: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:43.433734 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:43.433592 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:47.433577052 +0000 UTC m=+8.969719352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-klpkz" (UniqueName: "kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:44.126840 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:44.126810 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:44.127099 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:44.127083 5719 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:28:44.697257 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:44.697193 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerStarted","Data":"56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95"} Oct 31 00:28:44.697257 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:44.697256 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerStarted","Data":"bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db"} Oct 31 00:28:44.697685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:44.697282 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerStarted","Data":"65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350"} Oct 31 00:28:44.847353 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:44.847322 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:44.851415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:44.851391 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hxlqh" Oct 31 00:28:44.906589 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:44.906568 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vk9bb" Oct 31 00:28:45.297408 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:45.297377 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:45.703989 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:45.703951 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerStarted","Data":"602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e"} Oct 31 00:28:45.704355 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:45.704026 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerStarted","Data":"fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4"} Oct 31 00:28:45.704355 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:45.704055 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerStarted","Data":"4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8"} Oct 31 00:28:47.372610 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:47.372556 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:47.372610 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:47.372610 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:47.372664 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:47.372685 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372719 5719 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372786 5719 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:47.372762 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372824 5719 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372846 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:55.372828889 +0000 UTC m=+16.908971172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372869 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:55.37286231 +0000 UTC m=+16.909004593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"service-ca" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:47.372897 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372924 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:55.372900202 +0000 UTC m=+16.909042524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372928 5719 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372755 5719 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372998 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:55.372982433 +0000 UTC m=+16.909124792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-config" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.373024 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:55.373011267 +0000 UTC m=+16.909153551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.372824 5719 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:47.373173 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.373053 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:55.373047391 +0000 UTC m=+16.909189674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:47.473910 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:47.473864 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:47.474057 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.474014 5719 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:28:47.474057 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.474033 5719 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:28:47.474057 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.474045 5719 projected.go:196] Error preparing data for projected volume kube-api-access-klpkz for pod openshift-console/console-697bb4b4df-4gpsf: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:47.474147 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:47.474093 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:28:55.474081098 +0000 UTC m=+17.010223444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-klpkz" (UniqueName: "kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:50.443880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:50.443844 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:55.440538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:55.440499 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:55.440538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:55.440538 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:55.440575 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:55.440604 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440623 5719 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440662 5719 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:55.440662 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440729 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.440686362 +0000 UTC m=+32.976828668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440730 5719 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440759 5719 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440787 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.44076105 +0000 UTC m=+32.976903380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-serving-cert" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440797 5719 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440811 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.440799486 +0000 UTC m=+32.976941785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"oauth-serving-cert" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:55.440765 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440818 5719 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440823 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.440817442 +0000 UTC m=+32.976959725 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-oauth-config" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440886 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.440871247 +0000 UTC m=+32.977013529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"service-ca" not registered Oct 31 00:28:55.441016 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.440900 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.440894217 +0000 UTC m=+32.977036500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : object "openshift-console"/"console-config" not registered Oct 31 00:28:55.541379 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:55.541350 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"console-697bb4b4df-4gpsf\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:28:55.541512 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.541494 5719 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 31 00:28:55.541567 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.541518 5719 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 31 00:28:55.541567 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.541529 5719 projected.go:196] Error preparing data for projected volume kube-api-access-klpkz for pod openshift-console/console-697bb4b4df-4gpsf: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:55.541624 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:28:55.541568 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz podName:1368083a-8041-45bb-8e12-93f10e05eb15 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:11.541558459 +0000 UTC m=+33.077700741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-klpkz" (UniqueName: "kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz") pod "console-697bb4b4df-4gpsf" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 31 00:28:57.700795 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:57.700762 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:57.753711 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:57.753676 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:57.768609 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:57.768587 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:28:59.122421 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:59.122386 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:28:59.777860 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:28:59.777830 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m98mn" Oct 31 00:29:03.008143 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:03.008111 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:29:04.707391 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:04.707346 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-697bb4b4df-4gpsf" podUID="1368083a-8041-45bb-8e12-93f10e05eb15" containerName="console" containerID="cri-o://50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67" gracePeriod=15 Oct 31 00:29:04.983909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:04.983863 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:29:05.023894 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.023868 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") pod \"1368083a-8041-45bb-8e12-93f10e05eb15\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " Oct 31 00:29:05.024023 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.023923 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") pod \"1368083a-8041-45bb-8e12-93f10e05eb15\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " Oct 31 00:29:05.024023 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.023945 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") pod \"1368083a-8041-45bb-8e12-93f10e05eb15\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " Oct 31 00:29:05.024023 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.023976 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") pod \"1368083a-8041-45bb-8e12-93f10e05eb15\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " Oct 31 00:29:05.024023 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.023997 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") pod \"1368083a-8041-45bb-8e12-93f10e05eb15\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " Oct 31 00:29:05.024214 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024027 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") pod \"1368083a-8041-45bb-8e12-93f10e05eb15\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " Oct 31 00:29:05.024214 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024176 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") pod \"1368083a-8041-45bb-8e12-93f10e05eb15\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " Oct 31 00:29:05.024314 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024248 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp\") pod \"1368083a-8041-45bb-8e12-93f10e05eb15\" (UID: \"1368083a-8041-45bb-8e12-93f10e05eb15\") " Oct 31 00:29:05.024462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024397 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config" pod "1368083a-8041-45bb-8e12-93f10e05eb15" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:05.024462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024413 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca" pod "1368083a-8041-45bb-8e12-93f10e05eb15" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:05.024462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024423 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle" pod "1368083a-8041-45bb-8e12-93f10e05eb15" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:05.024976 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024635 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp" pod "1368083a-8041-45bb-8e12-93f10e05eb15" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:05.024976 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024659 5719 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-console-config\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:05.024976 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024679 5719 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:05.024976 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024712 5719 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-trusted-ca-bundle\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:05.024976 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.024768 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert" pod "1368083a-8041-45bb-8e12-93f10e05eb15" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:05.026427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.026404 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert" pod "1368083a-8041-45bb-8e12-93f10e05eb15" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:05.026604 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.026585 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config" pod "1368083a-8041-45bb-8e12-93f10e05eb15" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:05.026724 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.026626 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz" pod "1368083a-8041-45bb-8e12-93f10e05eb15" (UID: "1368083a-8041-45bb-8e12-93f10e05eb15"). InnerVolumeSpecName "kube-api-access-klpkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:05.125419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.125390 5719 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1368083a-8041-45bb-8e12-93f10e05eb15-oauth-serving-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:05.125419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.125411 5719 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1368083a-8041-45bb-8e12-93f10e05eb15-tmp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:05.125419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.125421 5719 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-serving-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:05.125599 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.125430 5719 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1368083a-8041-45bb-8e12-93f10e05eb15-console-oauth-config\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:05.125599 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.125438 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klpkz\" (UniqueName: \"kubernetes.io/projected/1368083a-8041-45bb-8e12-93f10e05eb15-kube-api-access-klpkz\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:05.302075 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.302028 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:29:05.305915 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.305893 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d46547967-pkq4f" Oct 31 00:29:05.778616 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.778584 5719 generic.go:356] "Generic (PLEG): container finished" podID="1368083a-8041-45bb-8e12-93f10e05eb15" containerID="50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67" exitCode=0 Oct 31 00:29:05.779060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.778628 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-697bb4b4df-4gpsf" Oct 31 00:29:05.779060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.778656 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697bb4b4df-4gpsf" event={"ID":"1368083a-8041-45bb-8e12-93f10e05eb15","Type":"ContainerDied","Data":"50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67"} Oct 31 00:29:05.779060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.778718 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-697bb4b4df-4gpsf" event={"ID":"1368083a-8041-45bb-8e12-93f10e05eb15","Type":"ContainerDied","Data":"a5854abe3749dd55c8aeb07c3d7894ed8a35e88d2fa28f56418efb86b5b309fd"} Oct 31 00:29:05.779060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.778742 5719 scope.go:122] "RemoveContainer" containerID="50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67" Oct 31 00:29:05.788060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.788040 5719 scope.go:122] "RemoveContainer" containerID="50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67" Oct 31 00:29:05.788354 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:05.788337 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67\": container with ID starting with 50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67 not found: ID does not exist" containerID="50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67" Oct 31 00:29:05.788413 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.788363 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67"} err="failed to get container status \"50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67\": rpc error: code = NotFound desc = could not find container \"50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67\": container with ID starting with 50e2750e0381bf2fa77fb767c23c41cad46c09ba4e818dc8649501740a763d67 not found: ID does not exist" Oct 31 00:29:05.939134 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.939109 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-697bb4b4df-4gpsf"] Oct 31 00:29:05.986875 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:05.986839 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-697bb4b4df-4gpsf"] Oct 31 00:29:06.599306 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:06.599273 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1368083a-8041-45bb-8e12-93f10e05eb15" path="/var/lib/kubelet/pods/1368083a-8041-45bb-8e12-93f10e05eb15/volumes" Oct 31 00:29:10.927637 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:10.927610 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gzjf9" Oct 31 00:29:12.746211 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:12.746175 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:12.746595 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:12.746449 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="alertmanager" containerID="cri-o://ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971" gracePeriod=120 Oct 31 00:29:12.746595 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:12.746555 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy-metric" containerID="cri-o://885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06" gracePeriod=120 Oct 31 00:29:12.746595 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:12.746556 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy" containerID="cri-o://fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293" gracePeriod=120 Oct 31 00:29:12.746778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:12.746556 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy-web" containerID="cri-o://31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2" gracePeriod=120 Oct 31 00:29:12.746778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:12.746569 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="config-reloader" containerID="cri-o://87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3" gracePeriod=120 Oct 31 00:29:12.746778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:12.746601 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="prom-label-proxy" containerID="cri-o://5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9" gracePeriod=120 Oct 31 00:29:13.809669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809636 5719 generic.go:356] "Generic (PLEG): container finished" podID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerID="5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9" exitCode=0 Oct 31 00:29:13.809669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809661 5719 generic.go:356] "Generic (PLEG): container finished" podID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerID="fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293" exitCode=0 Oct 31 00:29:13.809669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809667 5719 generic.go:356] "Generic (PLEG): container finished" podID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerID="31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2" exitCode=0 Oct 31 00:29:13.809669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809673 5719 generic.go:356] "Generic (PLEG): container finished" podID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerID="87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3" exitCode=0 Oct 31 00:29:13.809669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809678 5719 generic.go:356] "Generic (PLEG): container finished" podID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerID="ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971" exitCode=0 Oct 31 00:29:13.810260 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809719 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerDied","Data":"5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9"} Oct 31 00:29:13.810260 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809784 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerDied","Data":"fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293"} Oct 31 00:29:13.810260 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809811 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerDied","Data":"31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2"} Oct 31 00:29:13.810260 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809838 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerDied","Data":"87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3"} Oct 31 00:29:13.810260 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:13.809863 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerDied","Data":"ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971"} Oct 31 00:29:14.023207 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.023179 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:14.091214 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091195 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091349 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091227 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091349 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091251 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091349 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091281 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091349 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091318 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091564 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091356 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091564 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091394 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091676 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091592 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:14.091770 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091751 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091832 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091804 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091889 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091856 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091943 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091897 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.091943 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.091938 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.092219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.092159 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy\") pod \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\" (UID: \"6c01729b-d9cb-44ff-bcfd-3db050a479b8\") " Oct 31 00:29:14.092309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.092242 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:14.092480 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.092460 5719 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-main-db\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.092536 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.092488 5719 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.093037 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.092778 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:14.095370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.095253 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:14.095529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.095305 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "kube-api-access-56zs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:14.095529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.095322 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:14.095529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.095441 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:14.095838 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.095604 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:14.095888 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.095826 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:14.096106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.096081 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:14.096677 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.096654 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:14.100538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.100519 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:14.106890 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.106869 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config" pod "6c01729b-d9cb-44ff-bcfd-3db050a479b8" (UID: "6c01729b-d9cb-44ff-bcfd-3db050a479b8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:14.193577 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193554 5719 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-cluster-tls-config\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193577 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193576 5719 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193586 5719 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-out\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193595 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zs9\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-kube-api-access-56zs9\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193604 5719 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-web-config\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193613 5719 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-main-tls\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193623 5719 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193633 5719 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193642 5719 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c01729b-d9cb-44ff-bcfd-3db050a479b8-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193650 5719 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c01729b-d9cb-44ff-bcfd-3db050a479b8-tls-assets\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.193722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.193658 5719 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c01729b-d9cb-44ff-bcfd-3db050a479b8-metrics-client-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.821420 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.821389 5719 generic.go:356] "Generic (PLEG): container finished" podID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerID="885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06" exitCode=0 Oct 31 00:29:14.821909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.821427 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerDied","Data":"885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06"} Oct 31 00:29:14.821909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.821475 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:14.821909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.821509 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c01729b-d9cb-44ff-bcfd-3db050a479b8","Type":"ContainerDied","Data":"adf94de2be2239c94b6926e78a9413109c3c9e0e7ab2706bfa14317941079985"} Oct 31 00:29:14.821909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.821549 5719 scope.go:122] "RemoveContainer" containerID="5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9" Oct 31 00:29:14.829149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.829125 5719 scope.go:122] "RemoveContainer" containerID="885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06" Oct 31 00:29:14.835794 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.835777 5719 scope.go:122] "RemoveContainer" containerID="fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293" Oct 31 00:29:14.842096 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.842081 5719 scope.go:122] "RemoveContainer" containerID="31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2" Oct 31 00:29:14.848345 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.848328 5719 scope.go:122] "RemoveContainer" containerID="87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3" Oct 31 00:29:14.858397 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.858382 5719 scope.go:122] "RemoveContainer" containerID="ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971" Oct 31 00:29:14.864939 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.864924 5719 scope.go:122] "RemoveContainer" containerID="122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d" Oct 31 00:29:14.871311 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.871198 5719 scope.go:122] "RemoveContainer" containerID="5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9" Oct 31 00:29:14.871478 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:14.871460 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9\": container with ID starting with 5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9 not found: ID does not exist" containerID="5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9" Oct 31 00:29:14.871526 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.871488 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9"} err="failed to get container status \"5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9\": rpc error: code = NotFound desc = could not find container \"5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9\": container with ID starting with 5339f07418de5733e986c4c31ae655211742b446dde09869545454ec2182ceb9 not found: ID does not exist" Oct 31 00:29:14.871526 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.871505 5719 scope.go:122] "RemoveContainer" containerID="885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06" Oct 31 00:29:14.871767 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:14.871749 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06\": container with ID starting with 885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06 not found: ID does not exist" containerID="885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06" Oct 31 00:29:14.871834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.871772 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06"} err="failed to get container status \"885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06\": rpc error: code = NotFound desc = could not find container \"885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06\": container with ID starting with 885ee4115b542fdabadaec7a809088ca09851c9a9ee073e369a42cc205874d06 not found: ID does not exist" Oct 31 00:29:14.871834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.871786 5719 scope.go:122] "RemoveContainer" containerID="fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293" Oct 31 00:29:14.872018 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:14.871996 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293\": container with ID starting with fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293 not found: ID does not exist" containerID="fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293" Oct 31 00:29:14.872070 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872026 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293"} err="failed to get container status \"fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293\": rpc error: code = NotFound desc = could not find container \"fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293\": container with ID starting with fee594f2425177ef28073b1c845eda5deb6b1d841a97f7b42f1e697ca58b0293 not found: ID does not exist" Oct 31 00:29:14.872070 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872040 5719 scope.go:122] "RemoveContainer" containerID="31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2" Oct 31 00:29:14.872271 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:14.872255 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2\": container with ID starting with 31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2 not found: ID does not exist" containerID="31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2" Oct 31 00:29:14.872315 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872276 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2"} err="failed to get container status \"31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2\": rpc error: code = NotFound desc = could not find container \"31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2\": container with ID starting with 31e7ea1577c9f3dfd4eb7646ace172fce39b93dc434cdcf6daa7e65f213487a2 not found: ID does not exist" Oct 31 00:29:14.872315 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872291 5719 scope.go:122] "RemoveContainer" containerID="87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3" Oct 31 00:29:14.872515 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:14.872502 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3\": container with ID starting with 87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3 not found: ID does not exist" containerID="87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3" Oct 31 00:29:14.872559 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872518 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3"} err="failed to get container status \"87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3\": rpc error: code = NotFound desc = could not find container \"87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3\": container with ID starting with 87f2a928b87a43325457303140208251be6e7f5837fcf12d4a0a84db64012ce3 not found: ID does not exist" Oct 31 00:29:14.872559 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872530 5719 scope.go:122] "RemoveContainer" containerID="ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971" Oct 31 00:29:14.872743 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:14.872725 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971\": container with ID starting with ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971 not found: ID does not exist" containerID="ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971" Oct 31 00:29:14.872781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872748 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971"} err="failed to get container status \"ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971\": rpc error: code = NotFound desc = could not find container \"ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971\": container with ID starting with ca9fa66503e688baf24542ada957d4a48d8e552ae1a25c3536dbfc653653d971 not found: ID does not exist" Oct 31 00:29:14.872781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872764 5719 scope.go:122] "RemoveContainer" containerID="122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d" Oct 31 00:29:14.872989 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:14.872974 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d\": container with ID starting with 122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d not found: ID does not exist" containerID="122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d" Oct 31 00:29:14.873031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:14.872992 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d"} err="failed to get container status \"122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d\": rpc error: code = NotFound desc = could not find container \"122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d\": container with ID starting with 122d0860a97abea5f8df9906329afdaf7fe612f832079be92334e7d75d38d76d not found: ID does not exist" Oct 31 00:29:15.031104 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.031071 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.081037 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082292 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082741 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="init-config-reloader" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082755 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="init-config-reloader" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082771 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082791 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082812 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="prom-label-proxy" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082819 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="prom-label-proxy" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082841 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy-web" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082848 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy-web" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082871 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="alertmanager" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082877 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="alertmanager" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082887 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="config-reloader" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082893 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="config-reloader" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082907 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1368083a-8041-45bb-8e12-93f10e05eb15" containerName="console" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082913 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1368083a-8041-45bb-8e12-93f10e05eb15" containerName="console" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082922 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy-metric" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.082941 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy-metric" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.083022 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="init-config-reloader" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.083031 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.083042 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="prom-label-proxy" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.083051 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy-web" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.083059 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="alertmanager" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.083065 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="config-reloader" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.083074 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1368083a-8041-45bb-8e12-93f10e05eb15" containerName="console" Oct 31 00:29:15.084886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.083096 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" containerName="kube-rbac-proxy-metric" Oct 31 00:29:15.113395 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.113339 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.167114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167055 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:29:15.167114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167076 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:29:15.167114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167089 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:29:15.167114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167061 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:29:15.167114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167056 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:29:15.167460 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167056 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:29:15.167460 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167106 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:29:15.167460 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167168 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tjsbm\"" Oct 31 00:29:15.167460 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.167297 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:29:15.201429 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201408 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201441 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201458 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201546 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/218ebb1b-681e-49dc-b635-83704e5a17eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201745 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201602 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/218ebb1b-681e-49dc-b635-83704e5a17eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201745 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201635 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ebb1b-681e-49dc-b635-83704e5a17eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201745 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201736 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201783 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201805 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218ebb1b-681e-49dc-b635-83704e5a17eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201856 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.201908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201885 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-web-config\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.202112 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201953 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgkpf\" (UniqueName: \"kubernetes.io/projected/218ebb1b-681e-49dc-b635-83704e5a17eb-kube-api-access-rgkpf\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.202112 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.201974 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218ebb1b-681e-49dc-b635-83704e5a17eb-config-out\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.214361 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.214337 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:29:15.215142 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.215125 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:15.303079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303054 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303175 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303149 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303196 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218ebb1b-681e-49dc-b635-83704e5a17eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303224 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303249 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-web-config\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303334 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303292 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgkpf\" (UniqueName: \"kubernetes.io/projected/218ebb1b-681e-49dc-b635-83704e5a17eb-kube-api-access-rgkpf\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303334 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303315 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218ebb1b-681e-49dc-b635-83704e5a17eb-config-out\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303352 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303375 5719 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:29:15.303437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303387 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303412 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303447 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/218ebb1b-681e-49dc-b635-83704e5a17eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303470 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/218ebb1b-681e-49dc-b635-83704e5a17eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.303635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.303487 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ebb1b-681e-49dc-b635-83704e5a17eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.305148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.304477 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/218ebb1b-681e-49dc-b635-83704e5a17eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.305148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.305077 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/218ebb1b-681e-49dc-b635-83704e5a17eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.306072 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.306050 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ebb1b-681e-49dc-b635-83704e5a17eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.306671 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.306630 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.306671 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.306661 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.306918 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.306901 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.306974 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.306919 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218ebb1b-681e-49dc-b635-83704e5a17eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.307038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.307018 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218ebb1b-681e-49dc-b635-83704e5a17eb-config-out\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.307182 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.307157 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.307266 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.307247 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-web-config\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.308156 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.308136 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.308156 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.308145 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/218ebb1b-681e-49dc-b635-83704e5a17eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.362015 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.361994 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgkpf\" (UniqueName: \"kubernetes.io/projected/218ebb1b-681e-49dc-b635-83704e5a17eb-kube-api-access-rgkpf\") pod \"alertmanager-main-0\" (UID: \"218ebb1b-681e-49dc-b635-83704e5a17eb\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:15.522172 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:15.522123 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:29:16.038891 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.038861 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:29:16.045094 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:29:16.045068 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218ebb1b_681e_49dc_b635_83704e5a17eb.slice/crio-58e5ae912a84f7312ba663cc4756439dbba19cdf411a4e3ebb5f8e410f99f3be WatchSource:0}: Error finding container 58e5ae912a84f7312ba663cc4756439dbba19cdf411a4e3ebb5f8e410f99f3be: Status 404 returned error can't find the container with id 58e5ae912a84f7312ba663cc4756439dbba19cdf411a4e3ebb5f8e410f99f3be Oct 31 00:29:16.438965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.438940 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5597468789-lbx94"] Oct 31 00:29:16.441322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.441305 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.491834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.491814 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 00:29:16.491972 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.491951 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 00:29:16.492119 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.491989 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 00:29:16.492119 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.491995 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lsvt9\"" Oct 31 00:29:16.492119 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.492091 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 00:29:16.492257 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.492126 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 00:29:16.514233 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.514203 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.514330 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.514240 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-telemeter-client-tls\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.514403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.514348 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-secret-telemeter-client\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.514403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.514381 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-federate-client-tls\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.514477 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.514458 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-metrics-client-ca\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.514536 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.514521 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.514583 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.514555 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-serving-certs-ca-bundle\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.514680 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.514665 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpnx\" (UniqueName: \"kubernetes.io/projected/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-kube-api-access-bzpnx\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.540536 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.540508 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 00:29:16.541400 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.541382 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5597468789-lbx94"] Oct 31 00:29:16.599253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.599231 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c01729b-d9cb-44ff-bcfd-3db050a479b8" path="/var/lib/kubelet/pods/6c01729b-d9cb-44ff-bcfd-3db050a479b8/volumes" Oct 31 00:29:16.615573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.615548 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-metrics-client-ca\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.615652 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.615591 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.615742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.615730 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-serving-certs-ca-bundle\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.615805 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.615773 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpnx\" (UniqueName: \"kubernetes.io/projected/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-kube-api-access-bzpnx\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.615923 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.615890 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.616001 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.615943 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-telemeter-client-tls\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.616073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.616000 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-secret-telemeter-client\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.616073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.616040 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-federate-client-tls\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.616335 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.616318 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-metrics-client-ca\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.616412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.616377 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-serving-certs-ca-bundle\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.616970 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.616911 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.618277 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.618247 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.618378 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.618349 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-federate-client-tls\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.618431 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.618374 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-telemeter-client-tls\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.618586 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.618568 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-secret-telemeter-client\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.675194 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.675170 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpnx\" (UniqueName: \"kubernetes.io/projected/7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b-kube-api-access-bzpnx\") pod \"telemeter-client-5597468789-lbx94\" (UID: \"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b\") " pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.753991 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.753942 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5597468789-lbx94" Oct 31 00:29:16.836068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.836039 5719 generic.go:356] "Generic (PLEG): container finished" podID="218ebb1b-681e-49dc-b635-83704e5a17eb" containerID="b22d7d756aebedd1a98f964484dc06acac4a50ea4c6614f696311dee17a1c0a7" exitCode=0 Oct 31 00:29:16.836196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.836130 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"218ebb1b-681e-49dc-b635-83704e5a17eb","Type":"ContainerDied","Data":"b22d7d756aebedd1a98f964484dc06acac4a50ea4c6614f696311dee17a1c0a7"} Oct 31 00:29:16.836246 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:16.836216 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"218ebb1b-681e-49dc-b635-83704e5a17eb","Type":"ContainerStarted","Data":"58e5ae912a84f7312ba663cc4756439dbba19cdf411a4e3ebb5f8e410f99f3be"} Oct 31 00:29:17.178267 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.178245 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5597468789-lbx94"] Oct 31 00:29:17.183871 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:29:17.183841 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c37daf4_abfa_4b1d_a4ab_fd5d7eadd39b.slice/crio-74198cc618b0fe6a2736967fbf6d90acda79217db7812ace80e9abbe4ae66211 WatchSource:0}: Error finding container 74198cc618b0fe6a2736967fbf6d90acda79217db7812ace80e9abbe4ae66211: Status 404 returned error can't find the container with id 74198cc618b0fe6a2736967fbf6d90acda79217db7812ace80e9abbe4ae66211 Oct 31 00:29:17.844513 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.844462 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"218ebb1b-681e-49dc-b635-83704e5a17eb","Type":"ContainerStarted","Data":"c5ef7790a77c83c6013355b42f3886699268e94ab2c33fb7dd5618e6aea2dc7a"} Oct 31 00:29:17.844909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.844571 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"218ebb1b-681e-49dc-b635-83704e5a17eb","Type":"ContainerStarted","Data":"369a5df70f1a09c647ba76d5efeceeb8d27426e1dd2575747a3e944c822d946f"} Oct 31 00:29:17.844909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.844616 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"218ebb1b-681e-49dc-b635-83704e5a17eb","Type":"ContainerStarted","Data":"88bb94943ba8c90bf6f37fafadec3c31a6ab26e88362ceac7cfc0a78a8d12772"} Oct 31 00:29:17.844909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.844657 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"218ebb1b-681e-49dc-b635-83704e5a17eb","Type":"ContainerStarted","Data":"e5b0731375cc8e540ab34c6c6ac44153e6de2803fe44d21c0daf0ac6c68f0182"} Oct 31 00:29:17.844909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.844720 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"218ebb1b-681e-49dc-b635-83704e5a17eb","Type":"ContainerStarted","Data":"863463ea6ac03a666930f96ff7497190430ba0238322d44018c358c9db32406f"} Oct 31 00:29:17.844909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.844760 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"218ebb1b-681e-49dc-b635-83704e5a17eb","Type":"ContainerStarted","Data":"c1d13088bb10e4f31c86754f4e18cc7737d6471944061018306d3a3eaafc2c0c"} Oct 31 00:29:17.845803 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.845773 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5597468789-lbx94" event={"ID":"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b","Type":"ContainerStarted","Data":"74198cc618b0fe6a2736967fbf6d90acda79217db7812ace80e9abbe4ae66211"} Oct 31 00:29:17.971011 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:17.970953 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.956114535 podStartE2EDuration="2.970934058s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.819523ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:17.969948705 +0000 UTC m=+39.506091011" watchObservedRunningTime="2026-10-31 00:29:17.970934058 +0000 UTC m=+39.507076364" Oct 31 00:29:18.324661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.324631 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:18.325129 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.325036 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="config-reloader" containerID="cri-o://bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db" gracePeriod=600 Oct 31 00:29:18.325129 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.325032 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy-web" containerID="cri-o://4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8" gracePeriod=600 Oct 31 00:29:18.325129 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.325074 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy" containerID="cri-o://fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4" gracePeriod=600 Oct 31 00:29:18.325129 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.325043 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="thanos-sidecar" containerID="cri-o://56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95" gracePeriod=600 Oct 31 00:29:18.325343 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.325023 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="prometheus" containerID="cri-o://65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350" gracePeriod=600 Oct 31 00:29:18.325343 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.325213 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e" gracePeriod=600 Oct 31 00:29:18.852246 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.852217 5719 generic.go:356] "Generic (PLEG): container finished" podID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerID="602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e" exitCode=0 Oct 31 00:29:18.852246 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.852241 5719 generic.go:356] "Generic (PLEG): container finished" podID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerID="4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8" exitCode=0 Oct 31 00:29:18.852246 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.852247 5719 generic.go:356] "Generic (PLEG): container finished" podID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerID="56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95" exitCode=0 Oct 31 00:29:18.852246 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.852253 5719 generic.go:356] "Generic (PLEG): container finished" podID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerID="bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db" exitCode=0 Oct 31 00:29:18.852512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.852288 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerDied","Data":"602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e"} Oct 31 00:29:18.852512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.852346 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerDied","Data":"4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8"} Oct 31 00:29:18.852512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.852373 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerDied","Data":"56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95"} Oct 31 00:29:18.852512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:18.852395 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerDied","Data":"bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db"} Oct 31 00:29:19.859510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:19.859482 5719 generic.go:356] "Generic (PLEG): container finished" podID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerID="fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4" exitCode=0 Oct 31 00:29:19.859860 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:19.859533 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerDied","Data":"fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4"} Oct 31 00:29:20.865360 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:20.865326 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5597468789-lbx94" event={"ID":"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b","Type":"ContainerStarted","Data":"704b11a7bd3d4cfed5ff903ee6093b67aa27c9aa24e2285f1d211052d736cab9"} Oct 31 00:29:20.865683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:20.865380 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5597468789-lbx94" event={"ID":"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b","Type":"ContainerStarted","Data":"632025650e153290fae062b4aab60af815697406fa2d8041893cb5236486237a"} Oct 31 00:29:20.865683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:20.865398 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5597468789-lbx94" event={"ID":"7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b","Type":"ContainerStarted","Data":"c79fffd3be13080b267e00baa22ee2d4f3ea7a04064feae4d4fdd1eb0c2d05a9"} Oct 31 00:29:21.130965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:21.130920 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5597468789-lbx94" podStartSLOduration=1.6730600249999998 podStartE2EDuration="5.130907954s" totalImagesPullingTime="3.457847929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:21.083392506 +0000 UTC m=+42.619534811" watchObservedRunningTime="2026-10-31 00:29:21.130907954 +0000 UTC m=+42.667050255" Oct 31 00:29:22.754857 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:22.754804 5719 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 00:29:22.755381 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:22.754876 5719 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 00:29:22.755381 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:22.754905 5719 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 00:29:22.755381 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:22.754926 5719 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 31 00:29:24.577747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.577710 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:24.688352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688295 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688331 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688351 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688549 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688389 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688549 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688512 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688623 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688559 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688623 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688589 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688630 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688660 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688716 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688759 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688769 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:24.688958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688802 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688817 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:24.688958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688847 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688890 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688921 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.688958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688960 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.689308 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.688986 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.689308 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689013 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.689308 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689015 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:24.689308 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689041 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.689308 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689079 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db\") pod \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\" (UID: \"11796f3e-c26f-4453-81fc-b50c5ebfa01e\") " Oct 31 00:29:24.689308 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689083 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:24.689308 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689251 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:24.689664 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689415 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:24.689664 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689460 5719 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.689664 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689481 5719 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.689664 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689496 5719 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.689664 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689511 5719 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-metrics-client-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.689664 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.689525 5719 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.690711 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.690655 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:24.690824 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.690718 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:24.691327 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.691302 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.691906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.691877 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.693424 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.693399 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.693542 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.693513 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.694046 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.694026 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:24.694133 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.694099 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.694277 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.694251 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.694356 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.694201 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "kube-api-access-p4gq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:24.694709 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.694659 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.694843 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.694819 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.695449 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.695402 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:24.704785 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.704764 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config" pod "11796f3e-c26f-4453-81fc-b50c5ebfa01e" (UID: "11796f3e-c26f-4453-81fc-b50c5ebfa01e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:24.790266 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790244 5719 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-metrics-client-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790266 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790265 5719 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790276 5719 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790286 5719 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790295 5719 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790304 5719 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790313 5719 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11796f3e-c26f-4453-81fc-b50c5ebfa01e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790321 5719 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-config-out\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790330 5719 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-tls-assets\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790338 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4gq2\" (UniqueName: \"kubernetes.io/projected/11796f3e-c26f-4453-81fc-b50c5ebfa01e-kube-api-access-p4gq2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790346 5719 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-grpc-tls\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790355 5719 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-web-config\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790363 5719 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11796f3e-c26f-4453-81fc-b50c5ebfa01e-prometheus-k8s-db\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790371 5719 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.790382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.790380 5719 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11796f3e-c26f-4453-81fc-b50c5ebfa01e-secret-kube-rbac-proxy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:24.883683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.883645 5719 generic.go:356] "Generic (PLEG): container finished" podID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerID="65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350" exitCode=0 Oct 31 00:29:24.883869 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.883727 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerDied","Data":"65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350"} Oct 31 00:29:24.883869 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.883768 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:24.883869 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.883790 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11796f3e-c26f-4453-81fc-b50c5ebfa01e","Type":"ContainerDied","Data":"d48dfd49ce4b1e5cd6d20bfa6e1d12ee1f821af1f6a7f8b8d21a279656cca0fd"} Oct 31 00:29:24.883869 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.883824 5719 scope.go:122] "RemoveContainer" containerID="602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e" Oct 31 00:29:24.891363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.891347 5719 scope.go:122] "RemoveContainer" containerID="fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4" Oct 31 00:29:24.898535 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.898512 5719 scope.go:122] "RemoveContainer" containerID="4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8" Oct 31 00:29:24.905505 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.905491 5719 scope.go:122] "RemoveContainer" containerID="56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95" Oct 31 00:29:24.912233 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.912216 5719 scope.go:122] "RemoveContainer" containerID="bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db" Oct 31 00:29:24.920935 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.920851 5719 scope.go:122] "RemoveContainer" containerID="65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350" Oct 31 00:29:24.928360 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.928343 5719 scope.go:122] "RemoveContainer" containerID="6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9" Oct 31 00:29:24.934825 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.934803 5719 scope.go:122] "RemoveContainer" containerID="602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e" Oct 31 00:29:24.935086 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:24.935060 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e\": container with ID starting with 602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e not found: ID does not exist" containerID="602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e" Oct 31 00:29:24.935143 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.935094 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e"} err="failed to get container status \"602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e\": rpc error: code = NotFound desc = could not find container \"602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e\": container with ID starting with 602f397a17b549d612ab4a1b0ef4e22e644dc9e5fcea2a07e197b17033a8c81e not found: ID does not exist" Oct 31 00:29:24.935143 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.935113 5719 scope.go:122] "RemoveContainer" containerID="fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4" Oct 31 00:29:24.935382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:24.935361 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4\": container with ID starting with fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4 not found: ID does not exist" containerID="fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4" Oct 31 00:29:24.935419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.935393 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4"} err="failed to get container status \"fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4\": rpc error: code = NotFound desc = could not find container \"fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4\": container with ID starting with fe07406e40954e1e9e0549f330d98b7a4efc096809a06cba0b8c694eb76cb1b4 not found: ID does not exist" Oct 31 00:29:24.935419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.935414 5719 scope.go:122] "RemoveContainer" containerID="4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8" Oct 31 00:29:24.935643 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:24.935625 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8\": container with ID starting with 4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8 not found: ID does not exist" containerID="4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8" Oct 31 00:29:24.935678 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.935649 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8"} err="failed to get container status \"4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8\": rpc error: code = NotFound desc = could not find container \"4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8\": container with ID starting with 4950e793eeab57d12c5794ea674047e70025620bd813a2d93d64b5487238f8d8 not found: ID does not exist" Oct 31 00:29:24.935678 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.935667 5719 scope.go:122] "RemoveContainer" containerID="56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95" Oct 31 00:29:24.936045 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:24.935916 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95\": container with ID starting with 56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95 not found: ID does not exist" containerID="56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95" Oct 31 00:29:24.936045 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.935936 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95"} err="failed to get container status \"56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95\": rpc error: code = NotFound desc = could not find container \"56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95\": container with ID starting with 56fc58acd6f9139fb6168d7b23b1ef147535ccd7f1ee4ea1a35c4d9e851e9b95 not found: ID does not exist" Oct 31 00:29:24.936045 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.935951 5719 scope.go:122] "RemoveContainer" containerID="bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db" Oct 31 00:29:24.936175 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:24.936156 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db\": container with ID starting with bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db not found: ID does not exist" containerID="bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db" Oct 31 00:29:24.936209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.936174 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db"} err="failed to get container status \"bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db\": rpc error: code = NotFound desc = could not find container \"bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db\": container with ID starting with bc9e372d8c2398a5e74e39b03c6bed64c20c7c04da18b04151344c2dcf3791db not found: ID does not exist" Oct 31 00:29:24.936209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.936189 5719 scope.go:122] "RemoveContainer" containerID="65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350" Oct 31 00:29:24.936377 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:24.936362 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350\": container with ID starting with 65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350 not found: ID does not exist" containerID="65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350" Oct 31 00:29:24.936418 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.936381 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350"} err="failed to get container status \"65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350\": rpc error: code = NotFound desc = could not find container \"65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350\": container with ID starting with 65eeb3a7b9d80403b4f5f52f693c007255c2fc4379b99da328e121da1de27350 not found: ID does not exist" Oct 31 00:29:24.936418 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.936394 5719 scope.go:122] "RemoveContainer" containerID="6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9" Oct 31 00:29:24.936583 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:24.936566 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9\": container with ID starting with 6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9 not found: ID does not exist" containerID="6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9" Oct 31 00:29:24.936619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:24.936589 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9"} err="failed to get container status \"6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9\": rpc error: code = NotFound desc = could not find container \"6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9\": container with ID starting with 6f124f40c2c81f00f7caf01d35350e327ac73200110ec1fe9360825c8f02e7d9 not found: ID does not exist" Oct 31 00:29:25.045149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.045079 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:25.050433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.050412 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:25.091934 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.091908 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:25.092232 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092217 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy" Oct 31 00:29:25.092232 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092232 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092253 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="config-reloader" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092257 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="config-reloader" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092272 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="thanos-sidecar" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092276 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="thanos-sidecar" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092283 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="init-config-reloader" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092287 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="init-config-reloader" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092294 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy-web" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092298 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy-web" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092305 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="prometheus" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092309 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="prometheus" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092315 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy-thanos" Oct 31 00:29:25.092322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092319 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy-thanos" Oct 31 00:29:25.092653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092364 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="config-reloader" Oct 31 00:29:25.092653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092373 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="thanos-sidecar" Oct 31 00:29:25.092653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092379 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy-web" Oct 31 00:29:25.092653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092384 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="prometheus" Oct 31 00:29:25.092653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092388 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy-thanos" Oct 31 00:29:25.092653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.092394 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" containerName="kube-rbac-proxy" Oct 31 00:29:25.096219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.096204 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.147420 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147399 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:29:25.147522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147424 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:29:25.147522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147446 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:29:25.147626 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147593 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:29:25.147626 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147619 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:29:25.147750 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147656 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:29:25.147803 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147749 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:29:25.147891 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147874 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:29:25.147956 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147902 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:29:25.148009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147909 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:29:25.148009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147909 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:29:25.148009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.147920 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:29:25.148123 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.148014 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-djuj1hhvgpfmh\"" Oct 31 00:29:25.148170 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.148149 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9q7lj\"" Oct 31 00:29:25.193456 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193436 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193544 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193462 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193544 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193496 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193627 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193593 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193627 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193614 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193687 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193635 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193687 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193654 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193731 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193856 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193837 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193891 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193865 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193923 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193884 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193963 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193948 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.193999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.193986 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.194032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.194006 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.194032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.194022 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.194089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.194036 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-config\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.194089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.194053 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.194145 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.194132 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.194180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.194161 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5z5n\" (UniqueName: \"kubernetes.io/projected/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-kube-api-access-g5z5n\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.194210 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.194178 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.196268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.196253 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:29:25.196380 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.196365 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:29:25.197110 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.197095 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:25.295268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295210 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295241 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295264 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295502 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295382 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-config\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295502 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295409 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295502 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295434 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295502 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295457 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5z5n\" (UniqueName: \"kubernetes.io/projected/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-kube-api-access-g5z5n\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295502 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295476 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295515 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295539 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295586 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295610 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295620 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295647 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295680 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295746 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.295789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295777 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.296390 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295812 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.296390 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295838 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.296390 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295876 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.296390 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.295918 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.296590 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.296437 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.297048 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.296993 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.298908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.297737 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.298908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.298172 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.298908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.298355 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.298908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.298531 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.298908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.298660 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.298908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.298717 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.299633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.298974 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.299633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.299043 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.299633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.299283 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-config\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.299633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.299566 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.300041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.299930 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.300922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.300902 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.301068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.301042 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.301220 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.301203 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.301571 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.301556 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.302141 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.302124 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.359466 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.359447 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5z5n\" (UniqueName: \"kubernetes.io/projected/7166d6c5-eb2d-4661-9315-e6ecfee88bd2-kube-api-access-g5z5n\") pod \"prometheus-k8s-0\" (UID: \"7166d6c5-eb2d-4661-9315-e6ecfee88bd2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.415416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.415394 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:25.848114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.848081 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:29:25.854084 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:29:25.854058 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7166d6c5_eb2d_4661_9315_e6ecfee88bd2.slice/crio-d0263c70d4967cf73770b16382a39676d3f533b10dcb1dde0ff37941b1252a3b WatchSource:0}: Error finding container d0263c70d4967cf73770b16382a39676d3f533b10dcb1dde0ff37941b1252a3b: Status 404 returned error can't find the container with id d0263c70d4967cf73770b16382a39676d3f533b10dcb1dde0ff37941b1252a3b Oct 31 00:29:25.888430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:25.888400 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7166d6c5-eb2d-4661-9315-e6ecfee88bd2","Type":"ContainerStarted","Data":"d0263c70d4967cf73770b16382a39676d3f533b10dcb1dde0ff37941b1252a3b"} Oct 31 00:29:26.599195 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:26.599167 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11796f3e-c26f-4453-81fc-b50c5ebfa01e" path="/var/lib/kubelet/pods/11796f3e-c26f-4453-81fc-b50c5ebfa01e/volumes" Oct 31 00:29:26.893896 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:26.893839 5719 generic.go:356] "Generic (PLEG): container finished" podID="7166d6c5-eb2d-4661-9315-e6ecfee88bd2" containerID="ecfe2568547dbf7b187d8c496684ad97a426eeed762708de796903f5e1c041d2" exitCode=0 Oct 31 00:29:26.894231 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:26.893903 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7166d6c5-eb2d-4661-9315-e6ecfee88bd2","Type":"ContainerDied","Data":"ecfe2568547dbf7b187d8c496684ad97a426eeed762708de796903f5e1c041d2"} Oct 31 00:29:27.901908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:27.901862 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7166d6c5-eb2d-4661-9315-e6ecfee88bd2","Type":"ContainerStarted","Data":"068e5348d88a7ada774daa80ed0c17e2c191ff49db54fa81552d7d0aa93ddda9"} Oct 31 00:29:27.902416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:27.901982 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7166d6c5-eb2d-4661-9315-e6ecfee88bd2","Type":"ContainerStarted","Data":"565f14dba456862e27e0a1ddb051bab45e0f2af5caaac9f9693a9156fdebe304"} Oct 31 00:29:27.902416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:27.902030 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7166d6c5-eb2d-4661-9315-e6ecfee88bd2","Type":"ContainerStarted","Data":"1e6e0eaeab2236c7caf84f1c9effd330247370bbd19a22351516289260e347f6"} Oct 31 00:29:27.902416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:27.902077 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7166d6c5-eb2d-4661-9315-e6ecfee88bd2","Type":"ContainerStarted","Data":"4c57f1a97f43546424f223d690e8adc7c72a474bcf5190167a2a97668cbd2d0a"} Oct 31 00:29:27.902416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:27.902123 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7166d6c5-eb2d-4661-9315-e6ecfee88bd2","Type":"ContainerStarted","Data":"8f3965abe04a46ea1b0e7575656f6d148fb07509647727f27b2a9b6df3cfed36"} Oct 31 00:29:27.902416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:27.902163 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7166d6c5-eb2d-4661-9315-e6ecfee88bd2","Type":"ContainerStarted","Data":"dac7c4abb937ed370f85207bce2ce395da9292608548c0615cf7202b7e795d07"} Oct 31 00:29:28.033157 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:28.033100 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.016338131 podStartE2EDuration="3.033082202s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.744071ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:28.032545602 +0000 UTC m=+49.568687908" watchObservedRunningTime="2026-10-31 00:29:28.033082202 +0000 UTC m=+49.569224507" Oct 31 00:29:30.416268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:30.416232 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:33.010396 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:29:33.010358 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:29:39.043354 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:39.043267 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:39.059283 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:39.059261 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:29:39.962210 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:29:39.962180 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:30:03.009382 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:30:03.009348 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:30:33.009192 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:30:33.009152 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:31:03.008406 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:31:03.008368 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:31:33.009179 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:31:33.009135 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:32:03.009049 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:32:03.009008 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:32:32.094122 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.094090 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j"] Oct 31 00:32:32.097094 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.097074 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.145483 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.145460 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpc9v\" (UniqueName: \"kubernetes.io/projected/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-kube-api-access-fpc9v\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.145603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.145495 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.145603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.145537 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.195137 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.195114 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 00:32:32.195137 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.195121 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 00:32:32.195137 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.195123 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rs4k2\"" Oct 31 00:32:32.196151 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.196122 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j"] Oct 31 00:32:32.246254 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.246229 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.246363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.246286 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpc9v\" (UniqueName: \"kubernetes.io/projected/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-kube-api-access-fpc9v\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.246363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.246308 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.246614 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.246593 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.246676 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.246612 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.306825 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.306768 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpc9v\" (UniqueName: \"kubernetes.io/projected/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-kube-api-access-fpc9v\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.409887 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.409866 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:32.779366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:32.779335 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j"] Oct 31 00:32:32.785233 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:32:32.785207 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15a7f40_fcb6_4a7c_867e_bc2d28a800b2.slice/crio-75598533cf4ae84994d59c5791ad119f26902420cee3e2d2a0037bb9d0acdecb WatchSource:0}: Error finding container 75598533cf4ae84994d59c5791ad119f26902420cee3e2d2a0037bb9d0acdecb: Status 404 returned error can't find the container with id 75598533cf4ae84994d59c5791ad119f26902420cee3e2d2a0037bb9d0acdecb Oct 31 00:32:33.008566 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:32:33.008533 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:32:33.575083 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:33.575046 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" event={"ID":"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2","Type":"ContainerStarted","Data":"75598533cf4ae84994d59c5791ad119f26902420cee3e2d2a0037bb9d0acdecb"} Oct 31 00:32:40.604169 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:40.604115 5719 generic.go:356] "Generic (PLEG): container finished" podID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerID="634923b54bb548da91cb11e61d0cbe455669cddd8b0b5ab981f824996b706e86" exitCode=0 Oct 31 00:32:40.604624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:40.604217 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" event={"ID":"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2","Type":"ContainerDied","Data":"634923b54bb548da91cb11e61d0cbe455669cddd8b0b5ab981f824996b706e86"} Oct 31 00:32:45.623673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:45.623639 5719 generic.go:356] "Generic (PLEG): container finished" podID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerID="8aa4678b3602973540ff87a714bffa0f7e81cdf188037f1766cb62b1570a39d0" exitCode=0 Oct 31 00:32:45.624113 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:45.623735 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" event={"ID":"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2","Type":"ContainerDied","Data":"8aa4678b3602973540ff87a714bffa0f7e81cdf188037f1766cb62b1570a39d0"} Oct 31 00:32:54.658457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:54.658385 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" event={"ID":"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2","Type":"ContainerStarted","Data":"3bccae3561dc03f777ceccc7ce83430610028f4234133adbf3848d83019ee2e1"} Oct 31 00:32:54.772808 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:54.772764 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" podStartSLOduration=2.836515184 podStartE2EDuration="22.77275183s" totalImagesPullingTime="19.936236646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:54.772551131 +0000 UTC m=+256.308693437" watchObservedRunningTime="2026-10-31 00:32:54.77275183 +0000 UTC m=+256.308894135" Oct 31 00:32:55.670372 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:55.670341 5719 generic.go:356] "Generic (PLEG): container finished" podID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerID="3bccae3561dc03f777ceccc7ce83430610028f4234133adbf3848d83019ee2e1" exitCode=0 Oct 31 00:32:55.670775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:55.670418 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" event={"ID":"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2","Type":"ContainerDied","Data":"3bccae3561dc03f777ceccc7ce83430610028f4234133adbf3848d83019ee2e1"} Oct 31 00:32:56.815130 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:56.815108 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:32:56.955569 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:56.955515 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-bundle\") pod \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " Oct 31 00:32:56.955669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:56.955571 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-util\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-util\") pod \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " Oct 31 00:32:56.955669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:56.955616 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-kube-api-access-fpc9v\" (UniqueName: \"kubernetes.io/projected/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-kube-api-access-fpc9v\") pod \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\" (UID: \"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2\") " Oct 31 00:32:56.956116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:56.956090 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-bundle" pod "d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" (UID: "d15a7f40-fcb6-4a7c-867e-bc2d28a800b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:56.957904 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:56.957879 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-kube-api-access-fpc9v" pod "d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" (UID: "d15a7f40-fcb6-4a7c-867e-bc2d28a800b2"). InnerVolumeSpecName "kube-api-access-fpc9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:56.960817 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:56.960791 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-util" pod "d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" (UID: "d15a7f40-fcb6-4a7c-867e-bc2d28a800b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:57.056666 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:57.056640 5719 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-bundle\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:57.056666 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:57.056662 5719 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-util\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:57.056802 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:57.056672 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpc9v\" (UniqueName: \"kubernetes.io/projected/d15a7f40-fcb6-4a7c-867e-bc2d28a800b2-kube-api-access-fpc9v\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:57.678880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:57.678847 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" event={"ID":"d15a7f40-fcb6-4a7c-867e-bc2d28a800b2","Type":"ContainerDied","Data":"75598533cf4ae84994d59c5791ad119f26902420cee3e2d2a0037bb9d0acdecb"} Oct 31 00:32:57.679032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:57.678892 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75598533cf4ae84994d59c5791ad119f26902420cee3e2d2a0037bb9d0acdecb" Oct 31 00:32:57.679032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:32:57.678857 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cltd4j" Oct 31 00:33:03.011080 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:33:03.011035 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:33:04.069321 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.069291 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z"] Oct 31 00:33:04.069706 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.069609 5719 cpu_manager.go:414] "containerMap: removing container" podUID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerName="util" Oct 31 00:33:04.069706 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.069620 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerName="util" Oct 31 00:33:04.069706 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.069647 5719 cpu_manager.go:414] "containerMap: removing container" podUID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerName="extract" Oct 31 00:33:04.069706 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.069652 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerName="extract" Oct 31 00:33:04.069706 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.069659 5719 cpu_manager.go:414] "containerMap: removing container" podUID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerName="pull" Oct 31 00:33:04.069706 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.069664 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerName="pull" Oct 31 00:33:04.069900 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.069729 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15a7f40-fcb6-4a7c-867e-bc2d28a800b2" containerName="extract" Oct 31 00:33:04.072191 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.072175 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.181730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.181705 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 00:33:04.181730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.181714 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 00:33:04.181925 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.181799 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 00:33:04.181966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.181928 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 00:33:04.181966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.181930 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 00:33:04.181966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.181940 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-km8d9\"" Oct 31 00:33:04.208364 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.208343 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z"] Oct 31 00:33:04.212739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.212716 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52c8460d-f078-4d76-a57e-718e2ce2cca9-tmp\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.212827 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.212755 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52c8460d-f078-4d76-a57e-718e2ce2cca9-manifests\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.212827 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.212775 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.212903 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.212872 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.212934 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.212911 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-metrics-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.212972 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.212946 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxjlj\" (UniqueName: \"kubernetes.io/projected/52c8460d-f078-4d76-a57e-718e2ce2cca9-kube-api-access-vxjlj\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.314050 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.314025 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-metrics-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.314144 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.314068 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxjlj\" (UniqueName: \"kubernetes.io/projected/52c8460d-f078-4d76-a57e-718e2ce2cca9-kube-api-access-vxjlj\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.314144 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.314131 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52c8460d-f078-4d76-a57e-718e2ce2cca9-tmp\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.314304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.314163 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52c8460d-f078-4d76-a57e-718e2ce2cca9-manifests\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.314304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.314273 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.314408 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.314314 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.314610 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.314582 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52c8460d-f078-4d76-a57e-718e2ce2cca9-manifests\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.316504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.316481 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52c8460d-f078-4d76-a57e-718e2ce2cca9-tmp\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.316760 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.316743 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.316955 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.316933 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-metrics-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.317017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.316938 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52c8460d-f078-4d76-a57e-718e2ce2cca9-webhook-cert\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.376256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.376205 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxjlj\" (UniqueName: \"kubernetes.io/projected/52c8460d-f078-4d76-a57e-718e2ce2cca9-kube-api-access-vxjlj\") pod \"opendatahub-operator-controller-manager-c54754479-ctf8z\" (UID: \"52c8460d-f078-4d76-a57e-718e2ce2cca9\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.385060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.385044 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:04.875062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:04.875032 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z"] Oct 31 00:33:04.883653 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:04.883624 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c8460d_f078_4d76_a57e_718e2ce2cca9.slice/crio-c342ab7c3dff396a4a3f2bcbea3466cba1633bfa8c562b2a71e9a870e06881b5 WatchSource:0}: Error finding container c342ab7c3dff396a4a3f2bcbea3466cba1633bfa8c562b2a71e9a870e06881b5: Status 404 returned error can't find the container with id c342ab7c3dff396a4a3f2bcbea3466cba1633bfa8c562b2a71e9a870e06881b5 Oct 31 00:33:05.710577 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:05.710542 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" event={"ID":"52c8460d-f078-4d76-a57e-718e2ce2cca9","Type":"ContainerStarted","Data":"c342ab7c3dff396a4a3f2bcbea3466cba1633bfa8c562b2a71e9a870e06881b5"} Oct 31 00:33:08.725470 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:08.725404 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" event={"ID":"52c8460d-f078-4d76-a57e-718e2ce2cca9","Type":"ContainerStarted","Data":"38bffe250cddf693204890378122eda233b952b4a4fbaf0e5bfcb8c21974571a"} Oct 31 00:33:09.730552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:09.730518 5719 generic.go:356] "Generic (PLEG): container finished" podID="52c8460d-f078-4d76-a57e-718e2ce2cca9" containerID="38bffe250cddf693204890378122eda233b952b4a4fbaf0e5bfcb8c21974571a" exitCode=0 Oct 31 00:33:09.730973 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:09.730604 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" event={"ID":"52c8460d-f078-4d76-a57e-718e2ce2cca9","Type":"ContainerDied","Data":"38bffe250cddf693204890378122eda233b952b4a4fbaf0e5bfcb8c21974571a"} Oct 31 00:33:11.739796 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:11.739762 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" event={"ID":"52c8460d-f078-4d76-a57e-718e2ce2cca9","Type":"ContainerStarted","Data":"16ef88fd2b565944e7321f2cea1e97dc7f8f3befdd7cd4a5cb305614df8409b8"} Oct 31 00:33:11.740197 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:11.739997 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:12.017139 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:12.017039 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" podStartSLOduration=4.09837319 podStartE2EDuration="9.017022863s" totalImagesPullingTime="4.46412398s" totalInitContainerRuntime="454.525693ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:12.015973476 +0000 UTC m=+273.552115781" watchObservedRunningTime="2026-10-31 00:33:12.017022863 +0000 UTC m=+273.553165162" Oct 31 00:33:20.257818 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.257744 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj"] Oct 31 00:33:20.262780 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.262744 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" Oct 31 00:33:20.343992 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.343959 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ba28f7-10e9-416e-af2e-96aab72d95a6-tmp\") pod \"jobset-operator-674cd59dc5-rswdj\" (UID: \"16ba28f7-10e9-416e-af2e-96aab72d95a6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" Oct 31 00:33:20.344114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.344011 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tl5\" (UniqueName: \"kubernetes.io/projected/16ba28f7-10e9-416e-af2e-96aab72d95a6-kube-api-access-v2tl5\") pod \"jobset-operator-674cd59dc5-rswdj\" (UID: \"16ba28f7-10e9-416e-af2e-96aab72d95a6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" Oct 31 00:33:20.364578 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.364554 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-bbmvv\"" Oct 31 00:33:20.364721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.364568 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 00:33:20.364721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.364569 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:33:20.365583 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.365566 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj"] Oct 31 00:33:20.444952 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.444922 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tl5\" (UniqueName: \"kubernetes.io/projected/16ba28f7-10e9-416e-af2e-96aab72d95a6-kube-api-access-v2tl5\") pod \"jobset-operator-674cd59dc5-rswdj\" (UID: \"16ba28f7-10e9-416e-af2e-96aab72d95a6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" Oct 31 00:33:20.445053 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.445020 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ba28f7-10e9-416e-af2e-96aab72d95a6-tmp\") pod \"jobset-operator-674cd59dc5-rswdj\" (UID: \"16ba28f7-10e9-416e-af2e-96aab72d95a6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" Oct 31 00:33:20.445301 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.445286 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16ba28f7-10e9-416e-af2e-96aab72d95a6-tmp\") pod \"jobset-operator-674cd59dc5-rswdj\" (UID: \"16ba28f7-10e9-416e-af2e-96aab72d95a6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" Oct 31 00:33:20.503971 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.503949 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tl5\" (UniqueName: \"kubernetes.io/projected/16ba28f7-10e9-416e-af2e-96aab72d95a6-kube-api-access-v2tl5\") pod \"jobset-operator-674cd59dc5-rswdj\" (UID: \"16ba28f7-10e9-416e-af2e-96aab72d95a6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" Oct 31 00:33:20.589906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.589890 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" Oct 31 00:33:20.968237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:20.968160 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj"] Oct 31 00:33:20.974017 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:20.973991 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16ba28f7_10e9_416e_af2e_96aab72d95a6.slice/crio-65083ef152c06d88f0024de1651c3a679e9981160be9ba5266e982c0c8e26514 WatchSource:0}: Error finding container 65083ef152c06d88f0024de1651c3a679e9981160be9ba5266e982c0c8e26514: Status 404 returned error can't find the container with id 65083ef152c06d88f0024de1651c3a679e9981160be9ba5266e982c0c8e26514 Oct 31 00:33:21.779479 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:21.779436 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" event={"ID":"16ba28f7-10e9-416e-af2e-96aab72d95a6","Type":"ContainerStarted","Data":"65083ef152c06d88f0024de1651c3a679e9981160be9ba5266e982c0c8e26514"} Oct 31 00:33:22.745071 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:22.745046 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c54754479-ctf8z" Oct 31 00:33:26.739365 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.739330 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs"] Oct 31 00:33:26.742006 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.741986 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:26.798726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.798679 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9mlm7\"" Oct 31 00:33:26.798834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.798801 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9b0da5-5f1b-45ec-9470-78d4f2faffc8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vkpgs\" (UID: \"8f9b0da5-5f1b-45ec-9470-78d4f2faffc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:26.798893 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.798839 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssmr\" (UniqueName: \"kubernetes.io/projected/8f9b0da5-5f1b-45ec-9470-78d4f2faffc8-kube-api-access-tssmr\") pod \"cert-manager-webhook-5dbbbf75cc-vkpgs\" (UID: \"8f9b0da5-5f1b-45ec-9470-78d4f2faffc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:26.803704 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.801794 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 00:33:26.804480 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.804450 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 00:33:26.808926 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.808902 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs"] Oct 31 00:33:26.809628 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.809599 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" event={"ID":"16ba28f7-10e9-416e-af2e-96aab72d95a6","Type":"ContainerStarted","Data":"32a476b35250414ab0fb57887e700ad2740bd970142963dff82237194afbaf5e"} Oct 31 00:33:26.899355 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.899318 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssmr\" (UniqueName: \"kubernetes.io/projected/8f9b0da5-5f1b-45ec-9470-78d4f2faffc8-kube-api-access-tssmr\") pod \"cert-manager-webhook-5dbbbf75cc-vkpgs\" (UID: \"8f9b0da5-5f1b-45ec-9470-78d4f2faffc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:26.899546 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.899441 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9b0da5-5f1b-45ec-9470-78d4f2faffc8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vkpgs\" (UID: \"8f9b0da5-5f1b-45ec-9470-78d4f2faffc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:26.961491 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.961452 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f9b0da5-5f1b-45ec-9470-78d4f2faffc8-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vkpgs\" (UID: \"8f9b0da5-5f1b-45ec-9470-78d4f2faffc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:26.963975 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:26.963947 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssmr\" (UniqueName: \"kubernetes.io/projected/8f9b0da5-5f1b-45ec-9470-78d4f2faffc8-kube-api-access-tssmr\") pod \"cert-manager-webhook-5dbbbf75cc-vkpgs\" (UID: \"8f9b0da5-5f1b-45ec-9470-78d4f2faffc8\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:27.007665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:27.007588 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rswdj" podStartSLOduration=1.741160931 podStartE2EDuration="7.007575651s" totalImagesPullingTime="5.26641472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:27.006935246 +0000 UTC m=+288.543077551" watchObservedRunningTime="2026-10-31 00:33:27.007575651 +0000 UTC m=+288.543717956" Oct 31 00:33:27.052565 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:27.052540 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:27.466916 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:27.466884 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs"] Oct 31 00:33:27.483852 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:27.483823 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f9b0da5_5f1b_45ec_9470_78d4f2faffc8.slice/crio-2d16ff127c4c83799ed36b0962efe12bacbbd62ede4c368ca2a1b003866b98f5 WatchSource:0}: Error finding container 2d16ff127c4c83799ed36b0962efe12bacbbd62ede4c368ca2a1b003866b98f5: Status 404 returned error can't find the container with id 2d16ff127c4c83799ed36b0962efe12bacbbd62ede4c368ca2a1b003866b98f5 Oct 31 00:33:27.814453 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:27.814381 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" event={"ID":"8f9b0da5-5f1b-45ec-9470-78d4f2faffc8","Type":"ContainerStarted","Data":"2d16ff127c4c83799ed36b0962efe12bacbbd62ede4c368ca2a1b003866b98f5"} Oct 31 00:33:28.384391 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.384360 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r"] Oct 31 00:33:28.387679 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.387662 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" Oct 31 00:33:28.415502 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.415464 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvc7r\" (UID: \"c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" Oct 31 00:33:28.415613 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.415512 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljgl\" (UniqueName: \"kubernetes.io/projected/c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c-kube-api-access-tljgl\") pod \"cert-manager-cainjector-5c57bbd4fc-lvc7r\" (UID: \"c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" Oct 31 00:33:28.441876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.441855 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ljr6h\"" Oct 31 00:33:28.487497 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.487468 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r"] Oct 31 00:33:28.516406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.516384 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvc7r\" (UID: \"c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" Oct 31 00:33:28.516531 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.516418 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljgl\" (UniqueName: \"kubernetes.io/projected/c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c-kube-api-access-tljgl\") pod \"cert-manager-cainjector-5c57bbd4fc-lvc7r\" (UID: \"c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" Oct 31 00:33:28.576869 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.576850 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lvc7r\" (UID: \"c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" Oct 31 00:33:28.576975 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.576883 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljgl\" (UniqueName: \"kubernetes.io/projected/c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c-kube-api-access-tljgl\") pod \"cert-manager-cainjector-5c57bbd4fc-lvc7r\" (UID: \"c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" Oct 31 00:33:28.698416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:28.698361 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" Oct 31 00:33:29.115924 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:29.115894 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r"] Oct 31 00:33:29.121603 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:29.121578 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc20b18cc_b1c7_4a06_8364_1cfceb6e9b0c.slice/crio-cb3f20240d5c9f2170901cab911a6475647fe2093305aaff1f3969df8b52c899 WatchSource:0}: Error finding container cb3f20240d5c9f2170901cab911a6475647fe2093305aaff1f3969df8b52c899: Status 404 returned error can't find the container with id cb3f20240d5c9f2170901cab911a6475647fe2093305aaff1f3969df8b52c899 Oct 31 00:33:29.827037 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:29.826999 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" event={"ID":"c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c","Type":"ContainerStarted","Data":"cb3f20240d5c9f2170901cab911a6475647fe2093305aaff1f3969df8b52c899"} Oct 31 00:33:32.780471 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.779093 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx"] Oct 31 00:33:32.782976 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.782953 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:32.838436 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.838380 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx"] Oct 31 00:33:32.854457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.854425 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xgc\" (UniqueName: \"kubernetes.io/projected/371050b9-e409-4c0d-83f1-b198aef0d8fc-kube-api-access-48xgc\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:32.854603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.854485 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/371050b9-e409-4c0d-83f1-b198aef0d8fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:32.854603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.854530 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/371050b9-e409-4c0d-83f1-b198aef0d8fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:32.884854 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.884832 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 00:33:32.884992 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.884884 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8x6pf\"" Oct 31 00:33:32.884992 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.884905 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:33:32.884992 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.884957 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 00:33:32.884992 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.884969 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:33:32.955475 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.955438 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xgc\" (UniqueName: \"kubernetes.io/projected/371050b9-e409-4c0d-83f1-b198aef0d8fc-kube-api-access-48xgc\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:32.955636 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.955493 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/371050b9-e409-4c0d-83f1-b198aef0d8fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:32.955636 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.955537 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/371050b9-e409-4c0d-83f1-b198aef0d8fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:32.955791 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:33:32.955667 5719 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Oct 31 00:33:32.955791 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:33:32.955774 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/371050b9-e409-4c0d-83f1-b198aef0d8fc-plugin-serving-cert podName:371050b9-e409-4c0d-83f1-b198aef0d8fc nodeName:}" failed. No retries permitted until 2026-10-31 00:33:33.455751152 +0000 UTC m=+294.991893451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/371050b9-e409-4c0d-83f1-b198aef0d8fc-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-xq2lx" (UID: "371050b9-e409-4c0d-83f1-b198aef0d8fc") : secret "plugin-serving-cert" not found Oct 31 00:33:32.956389 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:32.956367 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/371050b9-e409-4c0d-83f1-b198aef0d8fc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:33.016403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:33.016373 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xgc\" (UniqueName: \"kubernetes.io/projected/371050b9-e409-4c0d-83f1-b198aef0d8fc-kube-api-access-48xgc\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:33.016576 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:33:33.016548 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:33:33.462130 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:33.462099 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/371050b9-e409-4c0d-83f1-b198aef0d8fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:33.464667 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:33.464646 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/371050b9-e409-4c0d-83f1-b198aef0d8fc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xq2lx\" (UID: \"371050b9-e409-4c0d-83f1-b198aef0d8fc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:33.697387 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:33.697362 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" Oct 31 00:33:33.848487 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:33.848447 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" event={"ID":"c20b18cc-b1c7-4a06-8364-1cfceb6e9b0c","Type":"ContainerStarted","Data":"5ec81226cbd0f08d5ae47b754e7e0cf0573a7457449118c5389ce4d8cf6991ed"} Oct 31 00:33:33.850074 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:33.850047 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" event={"ID":"8f9b0da5-5f1b-45ec-9470-78d4f2faffc8","Type":"ContainerStarted","Data":"d56636f42cca08514377cbed74d5d0ac8e04e7dded57d699d278b01e3b65cdd8"} Oct 31 00:33:33.850182 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:33.850171 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:33.960398 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:33.960321 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lvc7r" podStartSLOduration=1.521441275 podStartE2EDuration="5.960308978s" totalImagesPullingTime="4.438867703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:33.959215844 +0000 UTC m=+295.495358148" watchObservedRunningTime="2026-10-31 00:33:33.960308978 +0000 UTC m=+295.496451277" Oct 31 00:33:34.078928 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:34.078901 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx"] Oct 31 00:33:34.081916 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:34.081802 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" podStartSLOduration=2.213731773 podStartE2EDuration="8.081782422s" totalImagesPullingTime="5.868050649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:34.080532918 +0000 UTC m=+295.616675222" watchObservedRunningTime="2026-10-31 00:33:34.081782422 +0000 UTC m=+295.617924729" Oct 31 00:33:34.085942 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:34.085913 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod371050b9_e409_4c0d_83f1_b198aef0d8fc.slice/crio-8d922fd557db59ea15d0f8bd01a9d4c7bde7dfdb0de92ba0daf43fc7eb6a411b WatchSource:0}: Error finding container 8d922fd557db59ea15d0f8bd01a9d4c7bde7dfdb0de92ba0daf43fc7eb6a411b: Status 404 returned error can't find the container with id 8d922fd557db59ea15d0f8bd01a9d4c7bde7dfdb0de92ba0daf43fc7eb6a411b Oct 31 00:33:34.858546 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:34.858507 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" event={"ID":"371050b9-e409-4c0d-83f1-b198aef0d8fc","Type":"ContainerStarted","Data":"8d922fd557db59ea15d0f8bd01a9d4c7bde7dfdb0de92ba0daf43fc7eb6a411b"} Oct 31 00:33:35.801130 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:35.801097 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-7pvnx"] Oct 31 00:33:35.804605 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:35.804586 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7pvnx" Oct 31 00:33:35.885928 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:35.885887 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/865bc856-d2e8-4838-8d32-fe1149a81b66-bound-sa-token\") pod \"cert-manager-597b95cb84-7pvnx\" (UID: \"865bc856-d2e8-4838-8d32-fe1149a81b66\") " pod="cert-manager/cert-manager-597b95cb84-7pvnx" Oct 31 00:33:35.886273 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:35.885943 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfswf\" (UniqueName: \"kubernetes.io/projected/865bc856-d2e8-4838-8d32-fe1149a81b66-kube-api-access-hfswf\") pod \"cert-manager-597b95cb84-7pvnx\" (UID: \"865bc856-d2e8-4838-8d32-fe1149a81b66\") " pod="cert-manager/cert-manager-597b95cb84-7pvnx" Oct 31 00:33:35.903400 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:35.903357 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kzf2s\"" Oct 31 00:33:35.904537 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:35.904513 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7pvnx"] Oct 31 00:33:35.987285 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:35.987253 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfswf\" (UniqueName: \"kubernetes.io/projected/865bc856-d2e8-4838-8d32-fe1149a81b66-kube-api-access-hfswf\") pod \"cert-manager-597b95cb84-7pvnx\" (UID: \"865bc856-d2e8-4838-8d32-fe1149a81b66\") " pod="cert-manager/cert-manager-597b95cb84-7pvnx" Oct 31 00:33:35.987451 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:35.987393 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/865bc856-d2e8-4838-8d32-fe1149a81b66-bound-sa-token\") pod \"cert-manager-597b95cb84-7pvnx\" (UID: \"865bc856-d2e8-4838-8d32-fe1149a81b66\") " pod="cert-manager/cert-manager-597b95cb84-7pvnx" Oct 31 00:33:36.046458 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:36.046432 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/865bc856-d2e8-4838-8d32-fe1149a81b66-bound-sa-token\") pod \"cert-manager-597b95cb84-7pvnx\" (UID: \"865bc856-d2e8-4838-8d32-fe1149a81b66\") " pod="cert-manager/cert-manager-597b95cb84-7pvnx" Oct 31 00:33:36.046952 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:36.046927 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfswf\" (UniqueName: \"kubernetes.io/projected/865bc856-d2e8-4838-8d32-fe1149a81b66-kube-api-access-hfswf\") pod \"cert-manager-597b95cb84-7pvnx\" (UID: \"865bc856-d2e8-4838-8d32-fe1149a81b66\") " pod="cert-manager/cert-manager-597b95cb84-7pvnx" Oct 31 00:33:36.116434 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:36.116408 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-7pvnx" Oct 31 00:33:36.512811 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:36.512733 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-7pvnx"] Oct 31 00:33:36.519749 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:36.519723 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod865bc856_d2e8_4838_8d32_fe1149a81b66.slice/crio-e6d1dc30de5638aebc8b98658a9d709c72a964deb3de0a5f279afd64fee2c26e WatchSource:0}: Error finding container e6d1dc30de5638aebc8b98658a9d709c72a964deb3de0a5f279afd64fee2c26e: Status 404 returned error can't find the container with id e6d1dc30de5638aebc8b98658a9d709c72a964deb3de0a5f279afd64fee2c26e Oct 31 00:33:36.869632 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:36.869597 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7pvnx" event={"ID":"865bc856-d2e8-4838-8d32-fe1149a81b66","Type":"ContainerStarted","Data":"fd36e85977dd72026372e06a12e274cce3df43fceace1035258c9242065aae1c"} Oct 31 00:33:36.869632 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:36.869637 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-7pvnx" event={"ID":"865bc856-d2e8-4838-8d32-fe1149a81b66","Type":"ContainerStarted","Data":"e6d1dc30de5638aebc8b98658a9d709c72a964deb3de0a5f279afd64fee2c26e"} Oct 31 00:33:36.980861 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:36.980768 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-7pvnx" podStartSLOduration=1.98075184 podStartE2EDuration="1.98075184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:36.980644681 +0000 UTC m=+298.516786987" watchObservedRunningTime="2026-10-31 00:33:36.98075184 +0000 UTC m=+298.516894147" Oct 31 00:33:37.880594 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:37.880564 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67865c945c-w5smw"] Oct 31 00:33:37.883676 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:37.883651 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:37.971561 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:37.971531 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67865c945c-w5smw"] Oct 31 00:33:37.979780 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:37.979749 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 00:33:38.007334 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.007300 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40461298-1154-443e-bd2f-b0dbd0ed9537-tls-certs\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.007798 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.007452 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40461298-1154-443e-bd2f-b0dbd0ed9537-tmp\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.007798 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.007627 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9k6\" (UniqueName: \"kubernetes.io/projected/40461298-1154-443e-bd2f-b0dbd0ed9537-kube-api-access-ls9k6\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.015205 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.015181 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 00:33:38.059779 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.059750 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:33:38.061257 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.061215 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 00:33:38.061584 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.061566 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wp9n9\"" Oct 31 00:33:38.108607 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.108581 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9k6\" (UniqueName: \"kubernetes.io/projected/40461298-1154-443e-bd2f-b0dbd0ed9537-kube-api-access-ls9k6\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.108793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.108633 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40461298-1154-443e-bd2f-b0dbd0ed9537-tls-certs\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.108793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.108723 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40461298-1154-443e-bd2f-b0dbd0ed9537-tmp\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.111814 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.111789 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40461298-1154-443e-bd2f-b0dbd0ed9537-tls-certs\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.112932 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.112905 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40461298-1154-443e-bd2f-b0dbd0ed9537-tmp\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.180205 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.180128 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9k6\" (UniqueName: \"kubernetes.io/projected/40461298-1154-443e-bd2f-b0dbd0ed9537-kube-api-access-ls9k6\") pod \"kube-auth-proxy-67865c945c-w5smw\" (UID: \"40461298-1154-443e-bd2f-b0dbd0ed9537\") " pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.199085 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.199057 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" Oct 31 00:33:38.581859 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.581824 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67865c945c-w5smw"] Oct 31 00:33:38.588248 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:38.588223 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40461298_1154_443e_bd2f_b0dbd0ed9537.slice/crio-42f283acd66f30c57b6b92bf4a83c7f6b19a26a66d8ca7abf7634c8ff5475c00 WatchSource:0}: Error finding container 42f283acd66f30c57b6b92bf4a83c7f6b19a26a66d8ca7abf7634c8ff5475c00: Status 404 returned error can't find the container with id 42f283acd66f30c57b6b92bf4a83c7f6b19a26a66d8ca7abf7634c8ff5475c00 Oct 31 00:33:38.882201 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:38.881596 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" event={"ID":"40461298-1154-443e-bd2f-b0dbd0ed9537","Type":"ContainerStarted","Data":"42f283acd66f30c57b6b92bf4a83c7f6b19a26a66d8ca7abf7634c8ff5475c00"} Oct 31 00:33:39.862064 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:39.862035 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vkpgs" Oct 31 00:33:40.894828 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:40.894795 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" event={"ID":"371050b9-e409-4c0d-83f1-b198aef0d8fc","Type":"ContainerStarted","Data":"f790bb30bed5f3a46fc00f61bca513837b2ed1551cc34b557181c406e445bb4a"} Oct 31 00:33:41.008044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:41.007997 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xq2lx" podStartSLOduration=2.315992589 podStartE2EDuration="9.00798259s" totalImagesPullingTime="6.691990001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:41.006978139 +0000 UTC m=+302.543120444" watchObservedRunningTime="2026-10-31 00:33:41.00798259 +0000 UTC m=+302.544124894" Oct 31 00:33:41.910351 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:41.910318 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f2g2j"] Oct 31 00:33:41.924931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:41.923576 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:33:41.924931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:41.924825 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" Oct 31 00:33:41.931223 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:41.931197 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:41.968073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:41.968052 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f2g2j"] Oct 31 00:33:41.971268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:41.971248 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:33:42.014931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.014898 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 00:33:42.015040 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.014946 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-trzc2\"" Oct 31 00:33:42.056634 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.056605 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3ed9194-eec5-4d49-b8ae-e086c7079bb8-config-file\") pod \"limitador-limitador-7c9b87d574-zh575\" (UID: \"d3ed9194-eec5-4d49-b8ae-e086c7079bb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:42.056786 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.056714 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mjb\" (UniqueName: \"kubernetes.io/projected/d3ed9194-eec5-4d49-b8ae-e086c7079bb8-kube-api-access-g9mjb\") pod \"limitador-limitador-7c9b87d574-zh575\" (UID: \"d3ed9194-eec5-4d49-b8ae-e086c7079bb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:42.056877 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.056823 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6l4\" (UniqueName: \"kubernetes.io/projected/5affa900-8a4d-42af-a10d-53c34361c781-kube-api-access-wk6l4\") pod \"authorino-56c6c9ddd5-f2g2j\" (UID: \"5affa900-8a4d-42af-a10d-53c34361c781\") " pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" Oct 31 00:33:42.109979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.109957 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 00:33:42.157732 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.157709 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3ed9194-eec5-4d49-b8ae-e086c7079bb8-config-file\") pod \"limitador-limitador-7c9b87d574-zh575\" (UID: \"d3ed9194-eec5-4d49-b8ae-e086c7079bb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:42.157819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.157758 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mjb\" (UniqueName: \"kubernetes.io/projected/d3ed9194-eec5-4d49-b8ae-e086c7079bb8-kube-api-access-g9mjb\") pod \"limitador-limitador-7c9b87d574-zh575\" (UID: \"d3ed9194-eec5-4d49-b8ae-e086c7079bb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:42.157819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.157796 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6l4\" (UniqueName: \"kubernetes.io/projected/5affa900-8a4d-42af-a10d-53c34361c781-kube-api-access-wk6l4\") pod \"authorino-56c6c9ddd5-f2g2j\" (UID: \"5affa900-8a4d-42af-a10d-53c34361c781\") " pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" Oct 31 00:33:42.158265 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.158241 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3ed9194-eec5-4d49-b8ae-e086c7079bb8-config-file\") pod \"limitador-limitador-7c9b87d574-zh575\" (UID: \"d3ed9194-eec5-4d49-b8ae-e086c7079bb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:42.210057 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.210034 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:33:42.219603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.219581 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mjb\" (UniqueName: \"kubernetes.io/projected/d3ed9194-eec5-4d49-b8ae-e086c7079bb8-kube-api-access-g9mjb\") pod \"limitador-limitador-7c9b87d574-zh575\" (UID: \"d3ed9194-eec5-4d49-b8ae-e086c7079bb8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:42.219955 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.219937 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6l4\" (UniqueName: \"kubernetes.io/projected/5affa900-8a4d-42af-a10d-53c34361c781-kube-api-access-wk6l4\") pod \"authorino-56c6c9ddd5-f2g2j\" (UID: \"5affa900-8a4d-42af-a10d-53c34361c781\") " pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" Oct 31 00:33:42.249877 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.249858 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" Oct 31 00:33:42.257018 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.256992 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:42.625516 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.625484 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f2g2j"] Oct 31 00:33:42.631257 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:42.631233 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5affa900_8a4d_42af_a10d_53c34361c781.slice/crio-27575dfa6220c07f9f18172710e480339532a58594faee81f6ab844b62382fa1 WatchSource:0}: Error finding container 27575dfa6220c07f9f18172710e480339532a58594faee81f6ab844b62382fa1: Status 404 returned error can't find the container with id 27575dfa6220c07f9f18172710e480339532a58594faee81f6ab844b62382fa1 Oct 31 00:33:42.632603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.632587 5719 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:33:42.676522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.676492 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:33:42.681924 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:33:42.681903 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ed9194_eec5_4d49_b8ae_e086c7079bb8.slice/crio-334633b65c031d1b3ce04715bb99f1e714fc85237e76bae122f2de99dfe8c2e3 WatchSource:0}: Error finding container 334633b65c031d1b3ce04715bb99f1e714fc85237e76bae122f2de99dfe8c2e3: Status 404 returned error can't find the container with id 334633b65c031d1b3ce04715bb99f1e714fc85237e76bae122f2de99dfe8c2e3 Oct 31 00:33:42.906807 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.906731 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" event={"ID":"5affa900-8a4d-42af-a10d-53c34361c781","Type":"ContainerStarted","Data":"27575dfa6220c07f9f18172710e480339532a58594faee81f6ab844b62382fa1"} Oct 31 00:33:42.907955 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.907918 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" event={"ID":"d3ed9194-eec5-4d49-b8ae-e086c7079bb8","Type":"ContainerStarted","Data":"334633b65c031d1b3ce04715bb99f1e714fc85237e76bae122f2de99dfe8c2e3"} Oct 31 00:33:42.909322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:42.909298 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" event={"ID":"40461298-1154-443e-bd2f-b0dbd0ed9537","Type":"ContainerStarted","Data":"6ad18860b6cc64be4b039e7ad934fbe96f7a0b6ed0dbff7155f2ad44ff1b5389"} Oct 31 00:33:43.022148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:43.022102 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67865c945c-w5smw" podStartSLOduration=2.552807778 podStartE2EDuration="6.022089631s" totalImagesPullingTime="3.469281853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:43.020985356 +0000 UTC m=+304.557127660" watchObservedRunningTime="2026-10-31 00:33:43.022089631 +0000 UTC m=+304.558231935" Oct 31 00:33:46.932846 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:46.932808 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" event={"ID":"d3ed9194-eec5-4d49-b8ae-e086c7079bb8","Type":"ContainerStarted","Data":"d44eaa3a937b73930726c3b7fa638a61ba6e713e84d16fbe856ecbe09241fd22"} Oct 31 00:33:46.933288 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:46.932883 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:33:46.934332 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:46.934304 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" event={"ID":"5affa900-8a4d-42af-a10d-53c34361c781","Type":"ContainerStarted","Data":"f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658"} Oct 31 00:33:47.045440 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:47.045383 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" podStartSLOduration=2.226151985 podStartE2EDuration="6.045366757s" totalImagesPullingTime="3.819214772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:47.045054327 +0000 UTC m=+308.581196637" watchObservedRunningTime="2026-10-31 00:33:47.045366757 +0000 UTC m=+308.581509063" Oct 31 00:33:47.163228 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:47.163184 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" podStartSLOduration=2.439213328 podStartE2EDuration="6.163170485s" totalImagesPullingTime="3.723957157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:47.162656985 +0000 UTC m=+308.698799290" watchObservedRunningTime="2026-10-31 00:33:47.163170485 +0000 UTC m=+308.699312790" Oct 31 00:33:57.941831 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:33:57.941795 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-zh575" Oct 31 00:34:03.010633 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:34:03.010591 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:34:26.106479 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.106451 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66859cdfc8-cjbxt"] Oct 31 00:34:26.112323 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.112302 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.162665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.162643 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 00:34:26.162840 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.162822 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 00:34:26.162906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.162842 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7zs4p\"" Oct 31 00:34:26.210430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.210405 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 00:34:26.210674 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.210659 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 00:34:26.210779 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.210659 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 00:34:26.212134 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.212117 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66859cdfc8-cjbxt"] Oct 31 00:34:26.230770 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.230751 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-metrics-certs\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.230854 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.230802 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nh2\" (UniqueName: \"kubernetes.io/projected/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-kube-api-access-r8nh2\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.230896 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.230879 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-webhook-cert\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.332311 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.332291 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-metrics-certs\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.332415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.332336 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nh2\" (UniqueName: \"kubernetes.io/projected/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-kube-api-access-r8nh2\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.332415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.332362 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-webhook-cert\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.334839 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.334822 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-metrics-certs\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.334940 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.334925 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-webhook-cert\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.391276 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.391228 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nh2\" (UniqueName: \"kubernetes.io/projected/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-kube-api-access-r8nh2\") pod \"maas-controller-66859cdfc8-cjbxt\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.424107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.424089 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:26.796105 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:26.796039 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66859cdfc8-cjbxt"] Oct 31 00:34:26.802008 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:34:26.801984 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b92fbb_81e8_4d80_8fb5_bc46b89d4eed.slice/crio-20119b082659c2b11a2908047031f0479172b154e75678c8e35c2766a7cd77db WatchSource:0}: Error finding container 20119b082659c2b11a2908047031f0479172b154e75678c8e35c2766a7cd77db: Status 404 returned error can't find the container with id 20119b082659c2b11a2908047031f0479172b154e75678c8e35c2766a7cd77db Oct 31 00:34:27.113490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:27.113459 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" event={"ID":"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed","Type":"ContainerStarted","Data":"20119b082659c2b11a2908047031f0479172b154e75678c8e35c2766a7cd77db"} Oct 31 00:34:33.009649 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:34:33.009616 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:34:34.144909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:34.144878 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" event={"ID":"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed","Type":"ContainerStarted","Data":"6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71"} Oct 31 00:34:34.145293 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:34.144957 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:34.285118 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:34.285077 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" podStartSLOduration=1.021396282 podStartE2EDuration="8.285064519s" totalImagesPullingTime="7.263668237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:34.284394739 +0000 UTC m=+355.820537043" watchObservedRunningTime="2026-10-31 00:34:34.285064519 +0000 UTC m=+355.821206823" Oct 31 00:34:37.468296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.468263 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq"] Oct 31 00:34:37.470982 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.470962 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.568954 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.568927 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 00:34:37.569086 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.569004 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-chdxw\"" Oct 31 00:34:37.569457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.569436 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 00:34:37.569551 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.569455 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 00:34:37.570785 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.570767 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq"] Oct 31 00:34:37.629233 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.629211 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.629349 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.629244 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ac674f-79e0-434e-9c62-b8e6beb611f9-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.629349 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.629264 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p2n7\" (UniqueName: \"kubernetes.io/projected/23ac674f-79e0-434e-9c62-b8e6beb611f9-kube-api-access-6p2n7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.629466 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.629349 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.730386 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.730330 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.730386 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.730356 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ac674f-79e0-434e-9c62-b8e6beb611f9-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.730386 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.730373 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p2n7\" (UniqueName: \"kubernetes.io/projected/23ac674f-79e0-434e-9c62-b8e6beb611f9-kube-api-access-6p2n7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.730534 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.730403 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.730534 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:34:37.730451 5719 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 31 00:34:37.730534 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:34:37.730516 5719 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-proxy-source podName:23ac674f-79e0-434e-9c62-b8e6beb611f9 nodeName:}" failed. No retries permitted until 2026-10-31 00:34:38.230496 +0000 UTC m=+359.766638303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" (UID: "23ac674f-79e0-434e-9c62-b8e6beb611f9") : configmap "usage-tenancy-proxy-source" not found Oct 31 00:34:37.731016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.730999 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.732916 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.732899 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/23ac674f-79e0-434e-9c62-b8e6beb611f9-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:37.791819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:37.791796 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p2n7\" (UniqueName: \"kubernetes.io/projected/23ac674f-79e0-434e-9c62-b8e6beb611f9-kube-api-access-6p2n7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:38.234686 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:38.234647 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:38.235428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:38.235410 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/23ac674f-79e0-434e-9c62-b8e6beb611f9-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-5cdlq\" (UID: \"23ac674f-79e0-434e-9c62-b8e6beb611f9\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:38.383584 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:38.383556 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:38.750417 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:38.750388 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq"] Oct 31 00:34:38.756314 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:34:38.756283 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ac674f_79e0_434e_9c62_b8e6beb611f9.slice/crio-bcb6c67b35fed249dc52a993c18b689fb9f689a51d845dfe3c73f2f116376acf WatchSource:0}: Error finding container bcb6c67b35fed249dc52a993c18b689fb9f689a51d845dfe3c73f2f116376acf: Status 404 returned error can't find the container with id bcb6c67b35fed249dc52a993c18b689fb9f689a51d845dfe3c73f2f116376acf Oct 31 00:34:39.171352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:39.171318 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" event={"ID":"23ac674f-79e0-434e-9c62-b8e6beb611f9","Type":"ContainerStarted","Data":"bcb6c67b35fed249dc52a993c18b689fb9f689a51d845dfe3c73f2f116376acf"} Oct 31 00:34:45.152419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:45.152384 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:53.244720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:53.244672 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" event={"ID":"23ac674f-79e0-434e-9c62-b8e6beb611f9","Type":"ContainerStarted","Data":"a19d860239ab5188cccfa735f6be17151c98dc4a5eb73a179cf86b5a42e7f915"} Oct 31 00:34:53.245076 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:53.244839 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:34:53.358026 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:53.357984 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" podStartSLOduration=1.996550831 podStartE2EDuration="16.357972729s" totalImagesPullingTime="14.361421898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:53.357242672 +0000 UTC m=+374.893384977" watchObservedRunningTime="2026-10-31 00:34:53.357972729 +0000 UTC m=+374.894115033" Oct 31 00:34:57.648921 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.648892 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66859cdfc8-cjbxt"] Oct 31 00:34:57.649447 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.649074 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" podUID="80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" containerName="manager" containerID="cri-o://6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71" gracePeriod=30 Oct 31 00:34:57.891319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.891297 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:57.933220 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.933106 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-kube-api-access-r8nh2\" (UniqueName: \"kubernetes.io/projected/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-kube-api-access-r8nh2\") pod \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " Oct 31 00:34:57.933293 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.933244 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-metrics-certs\") pod \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " Oct 31 00:34:57.933334 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.933325 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-webhook-cert\") pod \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\" (UID: \"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed\") " Oct 31 00:34:57.935289 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.935267 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-metrics-certs" pod "80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" (UID: "80b92fbb-81e8-4d80-8fb5-bc46b89d4eed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:57.935378 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.935302 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-kube-api-access-r8nh2" pod "80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" (UID: "80b92fbb-81e8-4d80-8fb5-bc46b89d4eed"). InnerVolumeSpecName "kube-api-access-r8nh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:57.935418 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:57.935387 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-webhook-cert" pod "80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" (UID: "80b92fbb-81e8-4d80-8fb5-bc46b89d4eed"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:58.034651 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.034619 5719 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-webhook-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:58.034651 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.034650 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8nh2\" (UniqueName: \"kubernetes.io/projected/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-kube-api-access-r8nh2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:58.034850 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.034661 5719 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed-metrics-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:58.267444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.267372 5719 generic.go:356] "Generic (PLEG): container finished" podID="80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" containerID="6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71" exitCode=0 Oct 31 00:34:58.267444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.267414 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" event={"ID":"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed","Type":"ContainerDied","Data":"6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71"} Oct 31 00:34:58.267444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.267423 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" Oct 31 00:34:58.267657 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.267485 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66859cdfc8-cjbxt" event={"ID":"80b92fbb-81e8-4d80-8fb5-bc46b89d4eed","Type":"ContainerDied","Data":"20119b082659c2b11a2908047031f0479172b154e75678c8e35c2766a7cd77db"} Oct 31 00:34:58.267657 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.267510 5719 scope.go:122] "RemoveContainer" containerID="6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71" Oct 31 00:34:58.276789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.276772 5719 scope.go:122] "RemoveContainer" containerID="6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71" Oct 31 00:34:58.277052 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:34:58.277036 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71\": container with ID starting with 6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71 not found: ID does not exist" containerID="6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71" Oct 31 00:34:58.277089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.277060 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71"} err="failed to get container status \"6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71\": rpc error: code = NotFound desc = could not find container \"6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71\": container with ID starting with 6881ff326080af59cadeea46821143e6a165b747da6f1230067220c6a3bfcb71 not found: ID does not exist" Oct 31 00:34:58.510198 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.510169 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66859cdfc8-cjbxt"] Oct 31 00:34:58.517154 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.517134 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66859cdfc8-cjbxt"] Oct 31 00:34:58.601098 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:34:58.601077 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" path="/var/lib/kubelet/pods/80b92fbb-81e8-4d80-8fb5-bc46b89d4eed/volumes" Oct 31 00:35:03.009281 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:35:03.009248 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:35:04.255389 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:04.255360 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-5cdlq" Oct 31 00:35:12.671192 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.671163 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vjnhm"] Oct 31 00:35:12.671580 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.671527 5719 cpu_manager.go:414] "containerMap: removing container" podUID="80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" containerName="manager" Oct 31 00:35:12.671580 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.671537 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" containerName="manager" Oct 31 00:35:12.671664 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.671605 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b92fbb-81e8-4d80-8fb5-bc46b89d4eed" containerName="manager" Oct 31 00:35:12.676779 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.676759 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:12.730666 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.730638 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vjnhm"] Oct 31 00:35:12.764420 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.764396 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbgjd\" (UniqueName: \"kubernetes.io/projected/3ff32f5a-d8bd-42e6-a017-67721b1ccb53-kube-api-access-qbgjd\") pod \"kserve-controller-manager-776cc66d47-vjnhm\" (UID: \"3ff32f5a-d8bd-42e6-a017-67721b1ccb53\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:12.764538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.764466 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ff32f5a-d8bd-42e6-a017-67721b1ccb53-cert\") pod \"kserve-controller-manager-776cc66d47-vjnhm\" (UID: \"3ff32f5a-d8bd-42e6-a017-67721b1ccb53\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:12.776847 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.776824 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-95ldj\"" Oct 31 00:35:12.776961 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.776837 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 00:35:12.778277 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.778256 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j"] Oct 31 00:35:12.781239 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.781218 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:12.832236 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.832216 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 00:35:12.832236 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.832218 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9jgrf\"" Oct 31 00:35:12.838127 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.838108 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j"] Oct 31 00:35:12.865141 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.865117 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbgjd\" (UniqueName: \"kubernetes.io/projected/3ff32f5a-d8bd-42e6-a017-67721b1ccb53-kube-api-access-qbgjd\") pod \"kserve-controller-manager-776cc66d47-vjnhm\" (UID: \"3ff32f5a-d8bd-42e6-a017-67721b1ccb53\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:12.865240 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.865154 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djx5n\" (UniqueName: \"kubernetes.io/projected/4e63cb7c-0ebd-4712-a706-bebe2c63e303-kube-api-access-djx5n\") pod \"llmisvc-controller-manager-869d5dcc5d-vvv7j\" (UID: \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:12.865240 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.865196 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ff32f5a-d8bd-42e6-a017-67721b1ccb53-cert\") pod \"kserve-controller-manager-776cc66d47-vjnhm\" (UID: \"3ff32f5a-d8bd-42e6-a017-67721b1ccb53\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:12.865240 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.865219 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e63cb7c-0ebd-4712-a706-bebe2c63e303-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vvv7j\" (UID: \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:12.867685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.867663 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ff32f5a-d8bd-42e6-a017-67721b1ccb53-cert\") pod \"kserve-controller-manager-776cc66d47-vjnhm\" (UID: \"3ff32f5a-d8bd-42e6-a017-67721b1ccb53\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:12.924552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.924493 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbgjd\" (UniqueName: \"kubernetes.io/projected/3ff32f5a-d8bd-42e6-a017-67721b1ccb53-kube-api-access-qbgjd\") pod \"kserve-controller-manager-776cc66d47-vjnhm\" (UID: \"3ff32f5a-d8bd-42e6-a017-67721b1ccb53\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:12.966245 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.966219 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e63cb7c-0ebd-4712-a706-bebe2c63e303-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vvv7j\" (UID: \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:12.966364 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.966289 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djx5n\" (UniqueName: \"kubernetes.io/projected/4e63cb7c-0ebd-4712-a706-bebe2c63e303-kube-api-access-djx5n\") pod \"llmisvc-controller-manager-869d5dcc5d-vvv7j\" (UID: \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:12.968621 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.968597 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e63cb7c-0ebd-4712-a706-bebe2c63e303-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-vvv7j\" (UID: \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:12.984418 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.984391 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-zcp2r"] Oct 31 00:35:12.987056 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.987038 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:12.988098 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:12.988082 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:13.026303 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.026277 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djx5n\" (UniqueName: \"kubernetes.io/projected/4e63cb7c-0ebd-4712-a706-bebe2c63e303-kube-api-access-djx5n\") pod \"llmisvc-controller-manager-869d5dcc5d-vvv7j\" (UID: \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:13.037932 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.037838 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 00:35:13.037932 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.037913 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-d5ptx\"" Oct 31 00:35:13.038124 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.037984 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 00:35:13.046851 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.046824 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zcp2r"] Oct 31 00:35:13.067409 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.067380 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ba94f8-58b4-44db-8f31-984b32325ccb-metrics-certs\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.067543 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.067472 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85ba94f8-58b4-44db-8f31-984b32325ccb-cert\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.067650 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.067622 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svh5\" (UniqueName: \"kubernetes.io/projected/85ba94f8-58b4-44db-8f31-984b32325ccb-kube-api-access-7svh5\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.094759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.094740 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:13.168399 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.168369 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85ba94f8-58b4-44db-8f31-984b32325ccb-cert\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.168538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.168445 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svh5\" (UniqueName: \"kubernetes.io/projected/85ba94f8-58b4-44db-8f31-984b32325ccb-kube-api-access-7svh5\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.168609 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.168538 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ba94f8-58b4-44db-8f31-984b32325ccb-metrics-certs\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.171126 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.171100 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85ba94f8-58b4-44db-8f31-984b32325ccb-cert\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.171778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.171757 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85ba94f8-58b4-44db-8f31-984b32325ccb-metrics-certs\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.227522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.227470 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svh5\" (UniqueName: \"kubernetes.io/projected/85ba94f8-58b4-44db-8f31-984b32325ccb-kube-api-access-7svh5\") pod \"odh-model-controller-769875db56-zcp2r\" (UID: \"85ba94f8-58b4-44db-8f31-984b32325ccb\") " pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.299504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.299483 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:13.357821 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.357793 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vjnhm"] Oct 31 00:35:13.364502 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:35:13.364477 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff32f5a_d8bd_42e6_a017_67721b1ccb53.slice/crio-035c543c251c6cc1946957de6989631531c891785478adae79b2f90c3260f8a2 WatchSource:0}: Error finding container 035c543c251c6cc1946957de6989631531c891785478adae79b2f90c3260f8a2: Status 404 returned error can't find the container with id 035c543c251c6cc1946957de6989631531c891785478adae79b2f90c3260f8a2 Oct 31 00:35:13.465125 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.465095 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j"] Oct 31 00:35:13.471557 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:35:13.471524 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e63cb7c_0ebd_4712_a706_bebe2c63e303.slice/crio-5a4c05cf5e926df2968775176e57ef5ff17b0da2df3c4da951a24fa284819674 WatchSource:0}: Error finding container 5a4c05cf5e926df2968775176e57ef5ff17b0da2df3c4da951a24fa284819674: Status 404 returned error can't find the container with id 5a4c05cf5e926df2968775176e57ef5ff17b0da2df3c4da951a24fa284819674 Oct 31 00:35:13.666128 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:13.666103 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zcp2r"] Oct 31 00:35:13.671794 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:35:13.671767 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ba94f8_58b4_44db_8f31_984b32325ccb.slice/crio-fac37faf1b070f306f911aa28f61a899d9f3238df3a0d988167fd484e3a243b0 WatchSource:0}: Error finding container fac37faf1b070f306f911aa28f61a899d9f3238df3a0d988167fd484e3a243b0: Status 404 returned error can't find the container with id fac37faf1b070f306f911aa28f61a899d9f3238df3a0d988167fd484e3a243b0 Oct 31 00:35:14.346921 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:14.346880 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zcp2r" event={"ID":"85ba94f8-58b4-44db-8f31-984b32325ccb","Type":"ContainerStarted","Data":"fac37faf1b070f306f911aa28f61a899d9f3238df3a0d988167fd484e3a243b0"} Oct 31 00:35:14.348020 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:14.347995 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" event={"ID":"4e63cb7c-0ebd-4712-a706-bebe2c63e303","Type":"ContainerStarted","Data":"5a4c05cf5e926df2968775176e57ef5ff17b0da2df3c4da951a24fa284819674"} Oct 31 00:35:14.349079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:14.349056 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" event={"ID":"3ff32f5a-d8bd-42e6-a017-67721b1ccb53","Type":"ContainerStarted","Data":"035c543c251c6cc1946957de6989631531c891785478adae79b2f90c3260f8a2"} Oct 31 00:35:17.366465 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:17.366428 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" event={"ID":"4e63cb7c-0ebd-4712-a706-bebe2c63e303","Type":"ContainerStarted","Data":"31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8"} Oct 31 00:35:17.366978 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:17.366520 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:17.367960 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:17.367935 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" event={"ID":"3ff32f5a-d8bd-42e6-a017-67721b1ccb53","Type":"ContainerStarted","Data":"090cd63a9f118e2d83a2a8a8076e12c674576c2f0ecfb7d125a7d70ac85c5e92"} Oct 31 00:35:17.368097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:17.368066 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:35:17.479081 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:17.479027 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" podStartSLOduration=2.460607571 podStartE2EDuration="5.47900873s" totalImagesPullingTime="3.018401159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:17.478707429 +0000 UTC m=+399.014849726" watchObservedRunningTime="2026-10-31 00:35:17.47900873 +0000 UTC m=+399.015151036" Oct 31 00:35:17.590795 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:17.590746 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" podStartSLOduration=2.36173333 podStartE2EDuration="5.590733577s" totalImagesPullingTime="3.229000247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:17.589958158 +0000 UTC m=+399.126100464" watchObservedRunningTime="2026-10-31 00:35:17.590733577 +0000 UTC m=+399.126876105" Oct 31 00:35:19.380425 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:19.380395 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zcp2r" event={"ID":"85ba94f8-58b4-44db-8f31-984b32325ccb","Type":"ContainerStarted","Data":"fcf9cfb519b1514d50a352df8d90ff0f86bbf5ef1752664bd53dcaa91cdad12f"} Oct 31 00:35:19.380845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:19.380526 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:19.493679 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:19.493631 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-zcp2r" podStartSLOduration=2.251588156 podStartE2EDuration="7.493616142s" totalImagesPullingTime="5.242027986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:19.49268549 +0000 UTC m=+401.028827796" watchObservedRunningTime="2026-10-31 00:35:19.493616142 +0000 UTC m=+401.029758448" Oct 31 00:35:28.523392 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:28.523358 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:35:30.386807 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:30.386777 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-zcp2r" Oct 31 00:35:33.009528 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:35:33.009495 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:35:40.526520 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:40.526474 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f2g2j"] Oct 31 00:35:40.527219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:40.527166 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" podUID="5affa900-8a4d-42af-a10d-53c34361c781" containerName="authorino" containerID="cri-o://f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658" gracePeriod=30 Oct 31 00:35:40.774075 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:40.774053 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" Oct 31 00:35:40.804142 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:40.804085 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5affa900-8a4d-42af-a10d-53c34361c781-kube-api-access-wk6l4\" (UniqueName: \"kubernetes.io/projected/5affa900-8a4d-42af-a10d-53c34361c781-kube-api-access-wk6l4\") pod \"5affa900-8a4d-42af-a10d-53c34361c781\" (UID: \"5affa900-8a4d-42af-a10d-53c34361c781\") " Oct 31 00:35:40.806287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:40.806255 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5affa900-8a4d-42af-a10d-53c34361c781-kube-api-access-wk6l4" pod "5affa900-8a4d-42af-a10d-53c34361c781" (UID: "5affa900-8a4d-42af-a10d-53c34361c781"). InnerVolumeSpecName "kube-api-access-wk6l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:40.904901 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:40.904878 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk6l4\" (UniqueName: \"kubernetes.io/projected/5affa900-8a4d-42af-a10d-53c34361c781-kube-api-access-wk6l4\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:41.487360 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.487327 5719 generic.go:356] "Generic (PLEG): container finished" podID="5affa900-8a4d-42af-a10d-53c34361c781" containerID="f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658" exitCode=0 Oct 31 00:35:41.487545 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.487375 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" Oct 31 00:35:41.487545 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.487412 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" event={"ID":"5affa900-8a4d-42af-a10d-53c34361c781","Type":"ContainerDied","Data":"f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658"} Oct 31 00:35:41.487545 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.487462 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-f2g2j" event={"ID":"5affa900-8a4d-42af-a10d-53c34361c781","Type":"ContainerDied","Data":"27575dfa6220c07f9f18172710e480339532a58594faee81f6ab844b62382fa1"} Oct 31 00:35:41.487545 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.487480 5719 scope.go:122] "RemoveContainer" containerID="f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658" Oct 31 00:35:41.497229 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.497211 5719 scope.go:122] "RemoveContainer" containerID="f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658" Oct 31 00:35:41.497507 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:35:41.497490 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658\": container with ID starting with f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658 not found: ID does not exist" containerID="f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658" Oct 31 00:35:41.497555 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.497514 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658"} err="failed to get container status \"f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658\": rpc error: code = NotFound desc = could not find container \"f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658\": container with ID starting with f36f7e3ebea6c5ad8a6eb2cc571499205be976b26869aed878dd4307a1ee1658 not found: ID does not exist" Oct 31 00:35:41.643193 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.643172 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f2g2j"] Oct 31 00:35:41.647449 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:41.647431 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-f2g2j"] Oct 31 00:35:42.600663 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:42.600630 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5affa900-8a4d-42af-a10d-53c34361c781" path="/var/lib/kubelet/pods/5affa900-8a4d-42af-a10d-53c34361c781/volumes" Oct 31 00:35:48.374711 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:48.374663 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:35:48.377555 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:35:48.377530 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vjnhm" Oct 31 00:36:03.009310 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:36:03.009277 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:36:21.901867 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:21.901791 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn"] Oct 31 00:36:21.902237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:21.902177 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5affa900-8a4d-42af-a10d-53c34361c781" containerName="authorino" Oct 31 00:36:21.902237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:21.902188 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5affa900-8a4d-42af-a10d-53c34361c781" containerName="authorino" Oct 31 00:36:21.902312 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:21.902253 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5affa900-8a4d-42af-a10d-53c34361c781" containerName="authorino" Oct 31 00:36:21.904538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:21.904523 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:21.960609 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:21.960583 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn"] Oct 31 00:36:22.063834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:22.063797 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgnf\" (UniqueName: \"kubernetes.io/projected/6afc4a25-54af-4485-a899-ca98fc0f0944-kube-api-access-xqgnf\") pod \"llmisvc-controller-manager-6466c554d8-rmwbn\" (UID: \"6afc4a25-54af-4485-a899-ca98fc0f0944\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:22.064007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:22.063915 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6afc4a25-54af-4485-a899-ca98fc0f0944-cert\") pod \"llmisvc-controller-manager-6466c554d8-rmwbn\" (UID: \"6afc4a25-54af-4485-a899-ca98fc0f0944\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:22.164871 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:22.164787 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgnf\" (UniqueName: \"kubernetes.io/projected/6afc4a25-54af-4485-a899-ca98fc0f0944-kube-api-access-xqgnf\") pod \"llmisvc-controller-manager-6466c554d8-rmwbn\" (UID: \"6afc4a25-54af-4485-a899-ca98fc0f0944\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:22.164871 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:22.164845 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6afc4a25-54af-4485-a899-ca98fc0f0944-cert\") pod \"llmisvc-controller-manager-6466c554d8-rmwbn\" (UID: \"6afc4a25-54af-4485-a899-ca98fc0f0944\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:22.167363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:22.167336 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6afc4a25-54af-4485-a899-ca98fc0f0944-cert\") pod \"llmisvc-controller-manager-6466c554d8-rmwbn\" (UID: \"6afc4a25-54af-4485-a899-ca98fc0f0944\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:22.224332 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:22.224299 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgnf\" (UniqueName: \"kubernetes.io/projected/6afc4a25-54af-4485-a899-ca98fc0f0944-kube-api-access-xqgnf\") pod \"llmisvc-controller-manager-6466c554d8-rmwbn\" (UID: \"6afc4a25-54af-4485-a899-ca98fc0f0944\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:22.516433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:22.516356 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:22.884657 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:22.884626 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn"] Oct 31 00:36:22.890465 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:36:22.890438 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afc4a25_54af_4485_a899_ca98fc0f0944.slice/crio-943e263e6aff207be55da8884b80c2789de5e773ae3113d1f6ac35caeb2aa565 WatchSource:0}: Error finding container 943e263e6aff207be55da8884b80c2789de5e773ae3113d1f6ac35caeb2aa565: Status 404 returned error can't find the container with id 943e263e6aff207be55da8884b80c2789de5e773ae3113d1f6ac35caeb2aa565 Oct 31 00:36:23.678130 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:23.678096 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" event={"ID":"6afc4a25-54af-4485-a899-ca98fc0f0944","Type":"ContainerStarted","Data":"943e263e6aff207be55da8884b80c2789de5e773ae3113d1f6ac35caeb2aa565"} Oct 31 00:36:24.684873 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:24.684829 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" event={"ID":"6afc4a25-54af-4485-a899-ca98fc0f0944","Type":"ContainerStarted","Data":"c44f3ea3339e6522bff44a884f97bec509e12ddd7ce4998e14483c778484b512"} Oct 31 00:36:24.685338 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:24.684989 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:24.798256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:24.798195 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" podStartSLOduration=3.017702445 podStartE2EDuration="3.798177895s" totalImagesPullingTime="780.47545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:24.797110436 +0000 UTC m=+466.333252763" watchObservedRunningTime="2026-10-31 00:36:24.798177895 +0000 UTC m=+466.334320200" Oct 31 00:36:33.008747 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:36:33.008710 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:36:55.691078 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:55.691047 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rmwbn" Oct 31 00:36:55.853113 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:55.853079 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j"] Oct 31 00:36:55.853370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:55.853319 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" podUID="4e63cb7c-0ebd-4712-a706-bebe2c63e303" containerName="manager" containerID="cri-o://31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8" gracePeriod=30 Oct 31 00:36:56.102888 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.102867 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:36:56.248534 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.248461 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e63cb7c-0ebd-4712-a706-bebe2c63e303-kube-api-access-djx5n\" (UniqueName: \"kubernetes.io/projected/4e63cb7c-0ebd-4712-a706-bebe2c63e303-kube-api-access-djx5n\") pod \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\" (UID: \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\") " Oct 31 00:36:56.248677 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.248533 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e63cb7c-0ebd-4712-a706-bebe2c63e303-cert\" (UniqueName: \"kubernetes.io/secret/4e63cb7c-0ebd-4712-a706-bebe2c63e303-cert\") pod \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\" (UID: \"4e63cb7c-0ebd-4712-a706-bebe2c63e303\") " Oct 31 00:36:56.250630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.250604 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e63cb7c-0ebd-4712-a706-bebe2c63e303-cert" pod "4e63cb7c-0ebd-4712-a706-bebe2c63e303" (UID: "4e63cb7c-0ebd-4712-a706-bebe2c63e303"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:56.250775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.250606 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e63cb7c-0ebd-4712-a706-bebe2c63e303-kube-api-access-djx5n" pod "4e63cb7c-0ebd-4712-a706-bebe2c63e303" (UID: "4e63cb7c-0ebd-4712-a706-bebe2c63e303"). InnerVolumeSpecName "kube-api-access-djx5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:56.349628 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.349603 5719 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e63cb7c-0ebd-4712-a706-bebe2c63e303-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:56.349628 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.349627 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djx5n\" (UniqueName: \"kubernetes.io/projected/4e63cb7c-0ebd-4712-a706-bebe2c63e303-kube-api-access-djx5n\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:56.846096 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.846069 5719 generic.go:356] "Generic (PLEG): container finished" podID="4e63cb7c-0ebd-4712-a706-bebe2c63e303" containerID="31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8" exitCode=0 Oct 31 00:36:56.846463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.846111 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" event={"ID":"4e63cb7c-0ebd-4712-a706-bebe2c63e303","Type":"ContainerDied","Data":"31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8"} Oct 31 00:36:56.846463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.846116 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" Oct 31 00:36:56.846463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.846144 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j" event={"ID":"4e63cb7c-0ebd-4712-a706-bebe2c63e303","Type":"ContainerDied","Data":"5a4c05cf5e926df2968775176e57ef5ff17b0da2df3c4da951a24fa284819674"} Oct 31 00:36:56.846463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.846164 5719 scope.go:122] "RemoveContainer" containerID="31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8" Oct 31 00:36:56.854827 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.854650 5719 scope.go:122] "RemoveContainer" containerID="31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8" Oct 31 00:36:56.854906 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:36:56.854888 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8\": container with ID starting with 31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8 not found: ID does not exist" containerID="31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8" Oct 31 00:36:56.854954 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:56.854919 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8"} err="failed to get container status \"31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8\": rpc error: code = NotFound desc = could not find container \"31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8\": container with ID starting with 31fb3f58cec01f93bbe18e72a8d719ef2be93b7d7f223e25ce1f813760320bc8 not found: ID does not exist" Oct 31 00:36:57.003477 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:57.003426 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j"] Oct 31 00:36:57.051086 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:57.051067 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-vvv7j"] Oct 31 00:36:58.600406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:36:58.600373 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e63cb7c-0ebd-4712-a706-bebe2c63e303" path="/var/lib/kubelet/pods/4e63cb7c-0ebd-4712-a706-bebe2c63e303/volumes" Oct 31 00:37:03.009021 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:37:03.008987 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:37:33.010853 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:37:33.010821 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:37:42.117319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:37:42.117291 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:37:51.206052 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:37:51.205952 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:38:03.010198 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:38:03.010161 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:38:05.138305 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:05.138267 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:38:18.271472 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:18.271428 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:38:33.009367 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:38:33.009328 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:38:34.609976 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.609944 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527"] Oct 31 00:38:34.610444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.610321 5719 cpu_manager.go:414] "containerMap: removing container" podUID="4e63cb7c-0ebd-4712-a706-bebe2c63e303" containerName="manager" Oct 31 00:38:34.610444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.610331 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e63cb7c-0ebd-4712-a706-bebe2c63e303" containerName="manager" Oct 31 00:38:34.610444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.610410 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e63cb7c-0ebd-4712-a706-bebe2c63e303" containerName="manager" Oct 31 00:38:34.612866 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.612846 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.663037 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.663007 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 00:38:34.663180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.663061 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5wx8d\"" Oct 31 00:38:34.675451 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675426 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx79m\" (UniqueName: \"kubernetes.io/projected/db26ddf5-d18b-4d40-8107-a76902839346-kube-api-access-xx79m\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.675560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675463 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db26ddf5-d18b-4d40-8107-a76902839346-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.675560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675494 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.675641 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675626 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.675680 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675653 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db26ddf5-d18b-4d40-8107-a76902839346-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.675743 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675680 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db26ddf5-d18b-4d40-8107-a76902839346-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.675808 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675794 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.675845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675817 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.675845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.675837 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.712300 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.712275 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527"] Oct 31 00:38:34.777226 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777187 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db26ddf5-d18b-4d40-8107-a76902839346-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777244 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777288 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777306 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db26ddf5-d18b-4d40-8107-a76902839346-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777329 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db26ddf5-d18b-4d40-8107-a76902839346-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777588 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777372 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777588 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777400 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777588 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777426 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777588 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777475 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx79m\" (UniqueName: \"kubernetes.io/projected/db26ddf5-d18b-4d40-8107-a76902839346-kube-api-access-xx79m\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777843 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777725 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777843 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777807 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.777843 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777830 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.778003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.777987 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.778427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.778402 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db26ddf5-d18b-4d40-8107-a76902839346-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.779824 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.779804 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db26ddf5-d18b-4d40-8107-a76902839346-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.780045 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.780027 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db26ddf5-d18b-4d40-8107-a76902839346-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.836067 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.836044 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx79m\" (UniqueName: \"kubernetes.io/projected/db26ddf5-d18b-4d40-8107-a76902839346-kube-api-access-xx79m\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.836164 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.836044 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db26ddf5-d18b-4d40-8107-a76902839346-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-fk527\" (UID: \"db26ddf5-d18b-4d40-8107-a76902839346\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:34.924481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:34.924429 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:35.345352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:35.345312 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527"] Oct 31 00:38:35.351596 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:38:35.351561 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb26ddf5_d18b_4d40_8107_a76902839346.slice/crio-305281770dab520735bf3947baf687d00a612333944766bb2f77aa03216459d5 WatchSource:0}: Error finding container 305281770dab520735bf3947baf687d00a612333944766bb2f77aa03216459d5: Status 404 returned error can't find the container with id 305281770dab520735bf3947baf687d00a612333944766bb2f77aa03216459d5 Oct 31 00:38:36.300065 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:36.300034 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" event={"ID":"db26ddf5-d18b-4d40-8107-a76902839346","Type":"ContainerStarted","Data":"305281770dab520735bf3947baf687d00a612333944766bb2f77aa03216459d5"} Oct 31 00:38:39.369800 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:39.368929 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:39.369800 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:39.369013 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:39.369800 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:39.369055 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:40.325631 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:40.325536 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" event={"ID":"db26ddf5-d18b-4d40-8107-a76902839346","Type":"ContainerStarted","Data":"8d265276176845dd753a7d9629ae10aa2b16e08bf7b07bd016071fc8af635045"} Oct 31 00:38:40.492448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:40.492395 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" podStartSLOduration=2.477828869 podStartE2EDuration="6.492382691s" totalImagesPullingTime="4.014553822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:40.491364712 +0000 UTC m=+602.027507021" watchObservedRunningTime="2026-10-31 00:38:40.492382691 +0000 UTC m=+602.028524995" Oct 31 00:38:40.924805 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:40.924764 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:40.926259 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:40.926224 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fk527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:38:40.926390 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:40.926274 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" podUID="db26ddf5-d18b-4d40-8107-a76902839346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:38:41.925871 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:41.925838 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fk527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:38:41.926264 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:41.925898 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" podUID="db26ddf5-d18b-4d40-8107-a76902839346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:38:42.925051 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:42.925014 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fk527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:38:42.925202 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:42.925083 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" podUID="db26ddf5-d18b-4d40-8107-a76902839346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:38:43.925055 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:43.925015 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fk527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:38:43.925549 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:43.925075 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" podUID="db26ddf5-d18b-4d40-8107-a76902839346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:38:44.925395 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:44.925361 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fk527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:38:44.925801 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:44.925422 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" podUID="db26ddf5-d18b-4d40-8107-a76902839346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:38:45.925355 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:45.925315 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-fk527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:38:45.925571 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:45.925375 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" podUID="db26ddf5-d18b-4d40-8107-a76902839346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Oct 31 00:38:47.491071 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:47.491035 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:47.491523 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:47.491441 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:38:47.492252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:38:47.492228 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-fk527" Oct 31 00:39:03.008492 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:39:03.008457 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:39:32.414274 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.414242 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn"] Oct 31 00:39:32.417861 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.417839 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:32.473067 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.473040 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn"] Oct 31 00:39:32.474752 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.474730 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2ts\" (UniqueName: \"kubernetes.io/projected/60bc19a2-3a5a-4597-b027-a445598a64d5-kube-api-access-rr2ts\") pod \"payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn\" (UID: \"60bc19a2-3a5a-4597-b027-a445598a64d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:32.474861 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.474796 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bc19a2-3a5a-4597-b027-a445598a64d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn\" (UID: \"60bc19a2-3a5a-4597-b027-a445598a64d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:32.518871 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.518849 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-95fa4445-dockercfg-vm5b8\"" Oct 31 00:39:32.518871 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.518865 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-95fa4445\"" Oct 31 00:39:32.575739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.575678 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2ts\" (UniqueName: \"kubernetes.io/projected/60bc19a2-3a5a-4597-b027-a445598a64d5-kube-api-access-rr2ts\") pod \"payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn\" (UID: \"60bc19a2-3a5a-4597-b027-a445598a64d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:32.575900 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.575778 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bc19a2-3a5a-4597-b027-a445598a64d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn\" (UID: \"60bc19a2-3a5a-4597-b027-a445598a64d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:32.576541 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.576520 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bc19a2-3a5a-4597-b027-a445598a64d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn\" (UID: \"60bc19a2-3a5a-4597-b027-a445598a64d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:32.635314 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.635288 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2ts\" (UniqueName: \"kubernetes.io/projected/60bc19a2-3a5a-4597-b027-a445598a64d5-kube-api-access-rr2ts\") pod \"payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn\" (UID: \"60bc19a2-3a5a-4597-b027-a445598a64d5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:32.730354 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:32.730276 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:33.009898 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:39:33.009824 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:39:33.106462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:33.106431 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn"] Oct 31 00:39:33.112439 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:39:33.112409 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bc19a2_3a5a_4597_b027_a445598a64d5.slice/crio-d3dfde3cfb26cfd9557123eb4cf9f22664b66b760604ba89a46b58314462e925 WatchSource:0}: Error finding container d3dfde3cfb26cfd9557123eb4cf9f22664b66b760604ba89a46b58314462e925: Status 404 returned error can't find the container with id d3dfde3cfb26cfd9557123eb4cf9f22664b66b760604ba89a46b58314462e925 Oct 31 00:39:33.114447 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:33.114431 5719 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:39:33.579216 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:33.579178 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" event={"ID":"60bc19a2-3a5a-4597-b027-a445598a64d5","Type":"ContainerStarted","Data":"d3dfde3cfb26cfd9557123eb4cf9f22664b66b760604ba89a46b58314462e925"} Oct 31 00:39:34.045676 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:34.045598 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:39:39.609900 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:39.609855 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" event={"ID":"60bc19a2-3a5a-4597-b027-a445598a64d5","Type":"ContainerStarted","Data":"921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b"} Oct 31 00:39:39.610296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:39.609979 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:39.722940 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:39.722894 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" podStartSLOduration=1.978570939 podStartE2EDuration="7.72288141s" totalImagesPullingTime="5.744310471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:39.721479386 +0000 UTC m=+661.257621691" watchObservedRunningTime="2026-10-31 00:39:39.72288141 +0000 UTC m=+661.259023717" Oct 31 00:39:41.562458 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.562419 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6"] Oct 31 00:39:41.566176 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.566158 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:41.616500 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.616468 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2b9202f4\"" Oct 31 00:39:41.616756 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.616733 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2b9202f4-dockercfg-lnjw2\"" Oct 31 00:39:41.660732 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.660682 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljjb\" (UniqueName: \"kubernetes.io/projected/87411602-8f4e-4d0e-b93d-fcb560477f99-kube-api-access-zljjb\") pod \"payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6\" (UID: \"87411602-8f4e-4d0e-b93d-fcb560477f99\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:41.660870 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.660826 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87411602-8f4e-4d0e-b93d-fcb560477f99-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6\" (UID: \"87411602-8f4e-4d0e-b93d-fcb560477f99\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:41.665250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.665233 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6"] Oct 31 00:39:41.710685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.710655 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l"] Oct 31 00:39:41.714416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.714398 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:41.761821 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.761795 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87411602-8f4e-4d0e-b93d-fcb560477f99-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6\" (UID: \"87411602-8f4e-4d0e-b93d-fcb560477f99\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:41.761908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.761850 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljjb\" (UniqueName: \"kubernetes.io/projected/87411602-8f4e-4d0e-b93d-fcb560477f99-kube-api-access-zljjb\") pod \"payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6\" (UID: \"87411602-8f4e-4d0e-b93d-fcb560477f99\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:41.762503 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.762484 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87411602-8f4e-4d0e-b93d-fcb560477f99-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6\" (UID: \"87411602-8f4e-4d0e-b93d-fcb560477f99\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:41.769379 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.769358 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l"] Oct 31 00:39:41.820617 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.820593 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljjb\" (UniqueName: \"kubernetes.io/projected/87411602-8f4e-4d0e-b93d-fcb560477f99-kube-api-access-zljjb\") pod \"payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6\" (UID: \"87411602-8f4e-4d0e-b93d-fcb560477f99\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:41.862772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.862741 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-plugins-config-volume\") pod \"payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l\" (UID: \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\") " pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:41.862870 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.862795 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggb6p\" (UniqueName: \"kubernetes.io/projected/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-kube-api-access-ggb6p\") pod \"payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l\" (UID: \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\") " pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:41.878673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.878653 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:41.963420 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.963390 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-plugins-config-volume\") pod \"payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l\" (UID: \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\") " pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:41.963551 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.963452 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggb6p\" (UniqueName: \"kubernetes.io/projected/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-kube-api-access-ggb6p\") pod \"payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l\" (UID: \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\") " pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:41.964180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:41.964160 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-plugins-config-volume\") pod \"payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l\" (UID: \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\") " pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:42.022719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.022681 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggb6p\" (UniqueName: \"kubernetes.io/projected/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-kube-api-access-ggb6p\") pod \"payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l\" (UID: \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\") " pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:42.026512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.026492 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:42.245999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.245970 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6"] Oct 31 00:39:42.251264 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:39:42.251238 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87411602_8f4e_4d0e_b93d_fcb560477f99.slice/crio-6a89ad59d41f1580ee84f11ee2a6a8322ce9cf6b682186352dba60ef14bd7704 WatchSource:0}: Error finding container 6a89ad59d41f1580ee84f11ee2a6a8322ce9cf6b682186352dba60ef14bd7704: Status 404 returned error can't find the container with id 6a89ad59d41f1580ee84f11ee2a6a8322ce9cf6b682186352dba60ef14bd7704 Oct 31 00:39:42.395784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.395731 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l"] Oct 31 00:39:42.400837 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:39:42.400814 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c67b9b_d686_40ee_91b8_8a7f8b888ae2.slice/crio-7fa52423bfd95295f672b69f454fbde3978922bb8a1b46cf297f2eb0a6f5f8ba WatchSource:0}: Error finding container 7fa52423bfd95295f672b69f454fbde3978922bb8a1b46cf297f2eb0a6f5f8ba: Status 404 returned error can't find the container with id 7fa52423bfd95295f672b69f454fbde3978922bb8a1b46cf297f2eb0a6f5f8ba Oct 31 00:39:42.624762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.624725 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" event={"ID":"71c67b9b-d686-40ee-91b8-8a7f8b888ae2","Type":"ContainerStarted","Data":"84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da"} Oct 31 00:39:42.625190 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.624785 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" event={"ID":"71c67b9b-d686-40ee-91b8-8a7f8b888ae2","Type":"ContainerStarted","Data":"7fa52423bfd95295f672b69f454fbde3978922bb8a1b46cf297f2eb0a6f5f8ba"} Oct 31 00:39:42.625190 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.624815 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:39:42.626176 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.626153 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" event={"ID":"87411602-8f4e-4d0e-b93d-fcb560477f99","Type":"ContainerStarted","Data":"a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb"} Oct 31 00:39:42.626267 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.626203 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:42.626267 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.626219 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" event={"ID":"87411602-8f4e-4d0e-b93d-fcb560477f99","Type":"ContainerStarted","Data":"6a89ad59d41f1580ee84f11ee2a6a8322ce9cf6b682186352dba60ef14bd7704"} Oct 31 00:39:42.738769 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.738672 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" podStartSLOduration=1.738659185 podStartE2EDuration="1.738659185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:42.737216404 +0000 UTC m=+664.273358710" watchObservedRunningTime="2026-10-31 00:39:42.738659185 +0000 UTC m=+664.274801490" Oct 31 00:39:42.848804 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:42.848765 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" podStartSLOduration=1.848752223 podStartE2EDuration="1.848752223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:42.8480978 +0000 UTC m=+664.384240105" watchObservedRunningTime="2026-10-31 00:39:42.848752223 +0000 UTC m=+664.384894528" Oct 31 00:39:44.195104 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:44.195074 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:39:50.617236 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:50.617207 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:39:53.633546 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:53.633521 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:39:53.633964 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:39:53.633675 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:40:00.406476 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:00.406444 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:40:03.008866 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:40:03.008826 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:40:05.747835 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.747800 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2"] Oct 31 00:40:05.753032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.753014 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.763991 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.763968 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.764096 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.764021 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2xb\" (UniqueName: \"kubernetes.io/projected/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-kube-api-access-zh2xb\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.764135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.764109 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.808412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.808392 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:40:05.856679 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.856636 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:05.856809 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.856724 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:40:05.857923 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.857906 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2"] Oct 31 00:40:05.865134 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.865117 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.865213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.865163 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2xb\" (UniqueName: \"kubernetes.io/projected/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-kube-api-access-zh2xb\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.865213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.865201 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.865912 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.865894 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.867501 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.867482 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:05.909050 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.909030 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:40:05.924507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:05.924483 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2xb\" (UniqueName: \"kubernetes.io/projected/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-kube-api-access-zh2xb\") pod \"maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:06.064874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:06.064818 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:06.433340 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:06.433303 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2"] Oct 31 00:40:06.440000 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:40:06.439973 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750a46fe_ac62_4e99_8e80_6c0c9fd2cc3e.slice/crio-af94cf81597b1c0b7a4044f48b4165f456d3e425114b8e512509caa192856d73 WatchSource:0}: Error finding container af94cf81597b1c0b7a4044f48b4165f456d3e425114b8e512509caa192856d73: Status 404 returned error can't find the container with id af94cf81597b1c0b7a4044f48b4165f456d3e425114b8e512509caa192856d73 Oct 31 00:40:06.735674 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:06.735600 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" event={"ID":"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e","Type":"ContainerStarted","Data":"af94cf81597b1c0b7a4044f48b4165f456d3e425114b8e512509caa192856d73"} Oct 31 00:40:11.764902 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:11.764866 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" event={"ID":"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e","Type":"ContainerStarted","Data":"6d43b35d32b21506ce4927460d00610b43b330e24cf1f7c641a00e606d9c0c80"} Oct 31 00:40:11.876373 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:11.876331 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" podStartSLOduration=2.15624713 podStartE2EDuration="6.876318767s" totalImagesPullingTime="4.720071637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:11.875942865 +0000 UTC m=+693.412085168" watchObservedRunningTime="2026-10-31 00:40:11.876318767 +0000 UTC m=+693.412461072" Oct 31 00:40:12.770376 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:12.770338 5719 generic.go:356] "Generic (PLEG): container finished" podID="750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e" containerID="6d43b35d32b21506ce4927460d00610b43b330e24cf1f7c641a00e606d9c0c80" exitCode=0 Oct 31 00:40:12.770779 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:12.770417 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" event={"ID":"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e","Type":"ContainerDied","Data":"6d43b35d32b21506ce4927460d00610b43b330e24cf1f7c641a00e606d9c0c80"} Oct 31 00:40:13.906630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:13.906608 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:13.937717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:13.937679 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-xks-ca\") pod \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " Oct 31 00:40:13.937826 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:13.937741 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-kube-api-access-zh2xb\" (UniqueName: \"kubernetes.io/projected/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-kube-api-access-zh2xb\") pod \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " Oct 31 00:40:13.937826 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:13.937802 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-openshift-ca\") pod \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\" (UID: \"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e\") " Oct 31 00:40:13.938138 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:13.938113 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-openshift-ca" pod "750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e" (UID: "750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:13.939916 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:13.939891 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-kube-api-access-zh2xb" pod "750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e" (UID: "750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e"). InnerVolumeSpecName "kube-api-access-zh2xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:13.940004 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:13.939931 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-xks-ca" pod "750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e" (UID: "750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:14.038747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:14.038652 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:14.038747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:14.038676 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh2xb\" (UniqueName: \"kubernetes.io/projected/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-kube-api-access-zh2xb\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:14.038747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:14.038686 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:14.782378 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:14.782348 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" Oct 31 00:40:14.782512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:14.782382 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2" event={"ID":"750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e","Type":"ContainerDied","Data":"af94cf81597b1c0b7a4044f48b4165f456d3e425114b8e512509caa192856d73"} Oct 31 00:40:14.782512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:14.782419 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af94cf81597b1c0b7a4044f48b4165f456d3e425114b8e512509caa192856d73" Oct 31 00:40:21.409944 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.409910 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7"] Oct 31 00:40:21.410335 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.410321 5719 cpu_manager.go:414] "containerMap: removing container" podUID="750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e" containerName="revoke-keys" Oct 31 00:40:21.410378 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.410336 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e" containerName="revoke-keys" Oct 31 00:40:21.410419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.410410 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e" containerName="revoke-keys" Oct 31 00:40:21.413591 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.413571 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.468878 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.468852 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7"] Oct 31 00:40:21.503142 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.503115 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895nq\" (UniqueName: \"kubernetes.io/projected/a54550c0-76d7-4ddf-adde-d73cee3f09fb-kube-api-access-895nq\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.503242 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.503164 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.503282 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.503235 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.514438 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.514414 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:40:21.514672 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.514658 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:21.514753 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.514738 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:40:21.514803 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.514793 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:40:21.603912 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.603892 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895nq\" (UniqueName: \"kubernetes.io/projected/a54550c0-76d7-4ddf-adde-d73cee3f09fb-kube-api-access-895nq\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.604004 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.603941 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.604004 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.603967 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.604551 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.604531 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.606389 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.606370 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.662639 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.662587 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895nq\" (UniqueName: \"kubernetes.io/projected/a54550c0-76d7-4ddf-adde-d73cee3f09fb-kube-api-access-895nq\") pod \"maas-api-revoke-keys-e2e-ait-24d78330-vzzp7\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:21.725531 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:21.725509 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:22.094025 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:22.093996 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7"] Oct 31 00:40:22.100963 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:40:22.100934 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54550c0_76d7_4ddf_adde_d73cee3f09fb.slice/crio-e79a2869c4584f90b0bc2c7b8c6c9abba2763ff7a4277c0bc68f3a3b121c0497 WatchSource:0}: Error finding container e79a2869c4584f90b0bc2c7b8c6c9abba2763ff7a4277c0bc68f3a3b121c0497: Status 404 returned error can't find the container with id e79a2869c4584f90b0bc2c7b8c6c9abba2763ff7a4277c0bc68f3a3b121c0497 Oct 31 00:40:22.821581 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:22.821539 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" event={"ID":"a54550c0-76d7-4ddf-adde-d73cee3f09fb","Type":"ContainerStarted","Data":"6da962087ce384577f265b912ece629d3010e51f13270761f654e6ac0d0b4ffb"} Oct 31 00:40:22.821970 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:22.821590 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" event={"ID":"a54550c0-76d7-4ddf-adde-d73cee3f09fb","Type":"ContainerStarted","Data":"e79a2869c4584f90b0bc2c7b8c6c9abba2763ff7a4277c0bc68f3a3b121c0497"} Oct 31 00:40:22.934193 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:22.934141 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" podStartSLOduration=1.934127401 podStartE2EDuration="1.934127401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:22.933209076 +0000 UTC m=+704.469351392" watchObservedRunningTime="2026-10-31 00:40:22.934127401 +0000 UTC m=+704.470269706" Oct 31 00:40:23.827876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:23.827843 5719 generic.go:356] "Generic (PLEG): container finished" podID="a54550c0-76d7-4ddf-adde-d73cee3f09fb" containerID="6da962087ce384577f265b912ece629d3010e51f13270761f654e6ac0d0b4ffb" exitCode=0 Oct 31 00:40:23.828273 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:23.827900 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" event={"ID":"a54550c0-76d7-4ddf-adde-d73cee3f09fb","Type":"ContainerDied","Data":"6da962087ce384577f265b912ece629d3010e51f13270761f654e6ac0d0b4ffb"} Oct 31 00:40:24.975473 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:24.975448 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:25.034294 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.034264 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-openshift-ca\") pod \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " Oct 31 00:40:25.034428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.034309 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54550c0-76d7-4ddf-adde-d73cee3f09fb-kube-api-access-895nq\" (UniqueName: \"kubernetes.io/projected/a54550c0-76d7-4ddf-adde-d73cee3f09fb-kube-api-access-895nq\") pod \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " Oct 31 00:40:25.034493 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.034430 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-xks-ca\") pod \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\" (UID: \"a54550c0-76d7-4ddf-adde-d73cee3f09fb\") " Oct 31 00:40:25.034602 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.034569 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-openshift-ca" pod "a54550c0-76d7-4ddf-adde-d73cee3f09fb" (UID: "a54550c0-76d7-4ddf-adde-d73cee3f09fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:25.034755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.034723 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:25.036462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.036440 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54550c0-76d7-4ddf-adde-d73cee3f09fb-kube-api-access-895nq" pod "a54550c0-76d7-4ddf-adde-d73cee3f09fb" (UID: "a54550c0-76d7-4ddf-adde-d73cee3f09fb"). InnerVolumeSpecName "kube-api-access-895nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:25.036545 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.036509 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-xks-ca" pod "a54550c0-76d7-4ddf-adde-d73cee3f09fb" (UID: "a54550c0-76d7-4ddf-adde-d73cee3f09fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:25.135844 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.135774 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895nq\" (UniqueName: \"kubernetes.io/projected/a54550c0-76d7-4ddf-adde-d73cee3f09fb-kube-api-access-895nq\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:25.135844 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.135799 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54550c0-76d7-4ddf-adde-d73cee3f09fb-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:25.839504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.839477 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" Oct 31 00:40:25.839504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.839492 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7" event={"ID":"a54550c0-76d7-4ddf-adde-d73cee3f09fb","Type":"ContainerDied","Data":"e79a2869c4584f90b0bc2c7b8c6c9abba2763ff7a4277c0bc68f3a3b121c0497"} Oct 31 00:40:25.839739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:25.839542 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e79a2869c4584f90b0bc2c7b8c6c9abba2763ff7a4277c0bc68f3a3b121c0497" Oct 31 00:40:26.739401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.739373 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4"] Oct 31 00:40:26.739817 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.739803 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a54550c0-76d7-4ddf-adde-d73cee3f09fb" containerName="revoke-keys" Oct 31 00:40:26.739872 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.739818 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54550c0-76d7-4ddf-adde-d73cee3f09fb" containerName="revoke-keys" Oct 31 00:40:26.739905 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.739897 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54550c0-76d7-4ddf-adde-d73cee3f09fb" containerName="revoke-keys" Oct 31 00:40:26.742984 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.742966 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:26.797601 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.797578 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4"] Oct 31 00:40:26.843521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.843493 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:40:26.843521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.843509 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:40:26.843665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.843520 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:40:26.843665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.843549 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:40:26.849995 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.849959 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:26.850088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.850005 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t5rh\" (UniqueName: \"kubernetes.io/projected/388f9b15-0c6a-490e-8437-d9aa0f4283ef-kube-api-access-7t5rh\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:26.850160 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.850142 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:26.951241 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.951217 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:26.951352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.951250 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t5rh\" (UniqueName: \"kubernetes.io/projected/388f9b15-0c6a-490e-8437-d9aa0f4283ef-kube-api-access-7t5rh\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:26.951352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.951300 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:26.951858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.951834 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:26.953743 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:26.953726 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:27.010296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:27.010248 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t5rh\" (UniqueName: \"kubernetes.io/projected/388f9b15-0c6a-490e-8437-d9aa0f4283ef-kube-api-access-7t5rh\") pod \"maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:27.055842 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:27.055823 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:27.422502 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:27.422471 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4"] Oct 31 00:40:27.428679 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:40:27.428653 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388f9b15_0c6a_490e_8437_d9aa0f4283ef.slice/crio-02232727926a4c5843c7a259157faceeb351f49c67301769156231d23997ff60 WatchSource:0}: Error finding container 02232727926a4c5843c7a259157faceeb351f49c67301769156231d23997ff60: Status 404 returned error can't find the container with id 02232727926a4c5843c7a259157faceeb351f49c67301769156231d23997ff60 Oct 31 00:40:27.802108 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:27.802045 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2"] Oct 31 00:40:27.805174 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:27.805153 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6a47bf7-d5bm2"] Oct 31 00:40:27.851109 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:27.851079 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" event={"ID":"388f9b15-0c6a-490e-8437-d9aa0f4283ef","Type":"ContainerStarted","Data":"e40fc7f84123b34f22e54734211a10a5872e7bb700b22bb09515688480e659c7"} Oct 31 00:40:27.851210 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:27.851136 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" event={"ID":"388f9b15-0c6a-490e-8437-d9aa0f4283ef","Type":"ContainerStarted","Data":"02232727926a4c5843c7a259157faceeb351f49c67301769156231d23997ff60"} Oct 31 00:40:27.963743 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:27.963678 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" podStartSLOduration=1.9636637430000001 podStartE2EDuration="1.963663743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:27.96231619 +0000 UTC m=+709.498458495" watchObservedRunningTime="2026-10-31 00:40:27.963663743 +0000 UTC m=+709.499806047" Oct 31 00:40:28.601917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:28.601888 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e" path="/var/lib/kubelet/pods/750a46fe-ac62-4e99-8e80-6c0c9fd2cc3e/volumes" Oct 31 00:40:28.857686 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:28.857626 5719 generic.go:356] "Generic (PLEG): container finished" podID="388f9b15-0c6a-490e-8437-d9aa0f4283ef" containerID="e40fc7f84123b34f22e54734211a10a5872e7bb700b22bb09515688480e659c7" exitCode=0 Oct 31 00:40:28.858043 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:28.857727 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" event={"ID":"388f9b15-0c6a-490e-8437-d9aa0f4283ef","Type":"ContainerDied","Data":"e40fc7f84123b34f22e54734211a10a5872e7bb700b22bb09515688480e659c7"} Oct 31 00:40:30.001775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.001751 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:30.079223 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.079198 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-xks-ca\") pod \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " Oct 31 00:40:30.079388 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.079239 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-openshift-ca\") pod \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " Oct 31 00:40:30.079388 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.079288 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388f9b15-0c6a-490e-8437-d9aa0f4283ef-kube-api-access-7t5rh\" (UniqueName: \"kubernetes.io/projected/388f9b15-0c6a-490e-8437-d9aa0f4283ef-kube-api-access-7t5rh\") pod \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\" (UID: \"388f9b15-0c6a-490e-8437-d9aa0f4283ef\") " Oct 31 00:40:30.079810 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.079784 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-openshift-ca" pod "388f9b15-0c6a-490e-8437-d9aa0f4283ef" (UID: "388f9b15-0c6a-490e-8437-d9aa0f4283ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:30.081430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.081402 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-xks-ca" pod "388f9b15-0c6a-490e-8437-d9aa0f4283ef" (UID: "388f9b15-0c6a-490e-8437-d9aa0f4283ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:40:30.081617 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.081597 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388f9b15-0c6a-490e-8437-d9aa0f4283ef-kube-api-access-7t5rh" pod "388f9b15-0c6a-490e-8437-d9aa0f4283ef" (UID: "388f9b15-0c6a-490e-8437-d9aa0f4283ef"). InnerVolumeSpecName "kube-api-access-7t5rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:30.180495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.180421 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:30.180495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.180447 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/388f9b15-0c6a-490e-8437-d9aa0f4283ef-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:30.180495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.180457 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t5rh\" (UniqueName: \"kubernetes.io/projected/388f9b15-0c6a-490e-8437-d9aa0f4283ef-kube-api-access-7t5rh\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:30.868981 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.868906 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" Oct 31 00:40:30.868981 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.868930 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4" event={"ID":"388f9b15-0c6a-490e-8437-d9aa0f4283ef","Type":"ContainerDied","Data":"02232727926a4c5843c7a259157faceeb351f49c67301769156231d23997ff60"} Oct 31 00:40:30.868981 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:30.868969 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02232727926a4c5843c7a259157faceeb351f49c67301769156231d23997ff60" Oct 31 00:40:32.246669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:32.246633 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:40:33.008980 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:40:33.008944 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:40:38.402908 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:38.402874 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l"] Oct 31 00:40:38.449152 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:38.449126 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6"] Oct 31 00:40:39.147613 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:39.147577 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7"] Oct 31 00:40:39.151225 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:39.151200 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24d78330-vzzp7"] Oct 31 00:40:40.601196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:40.601156 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54550c0-76d7-4ddf-adde-d73cee3f09fb" path="/var/lib/kubelet/pods/a54550c0-76d7-4ddf-adde-d73cee3f09fb/volumes" Oct 31 00:40:41.325234 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:41.325198 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn"] Oct 31 00:40:42.164109 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:42.164071 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4"] Oct 31 00:40:42.167023 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:42.166999 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b9202f4-ldhz4"] Oct 31 00:40:42.601764 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:42.601732 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388f9b15-0c6a-490e-8437-d9aa0f4283ef" path="/var/lib/kubelet/pods/388f9b15-0c6a-490e-8437-d9aa0f4283ef/volumes" Oct 31 00:40:43.403313 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.403250 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" podUID="71c67b9b-d686-40ee-91b8-8a7f8b888ae2" containerName="payload-processing" containerID="cri-o://84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da" gracePeriod=25 Oct 31 00:40:43.450131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.450065 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" podUID="87411602-8f4e-4d0e-b93d-fcb560477f99" containerName="payload-pre-processing" containerID="cri-o://a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb" gracePeriod=25 Oct 31 00:40:43.631611 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:40:43.631549 5719 logging.go:55] [core] [Channel #68 SubChannel #69]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:50052", ServerName: "10.133.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:50052: connect: connection refused" Oct 31 00:40:43.707229 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.707202 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:40:43.730511 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.730361 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:40:43.899494 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.899463 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-plugins-config-volume\") pod \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\" (UID: \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\") " Oct 31 00:40:43.899494 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.899496 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-kube-api-access-ggb6p\" (UniqueName: \"kubernetes.io/projected/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-kube-api-access-ggb6p\") pod \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\" (UID: \"71c67b9b-d686-40ee-91b8-8a7f8b888ae2\") " Oct 31 00:40:43.899779 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.899527 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87411602-8f4e-4d0e-b93d-fcb560477f99-kube-api-access-zljjb\" (UniqueName: \"kubernetes.io/projected/87411602-8f4e-4d0e-b93d-fcb560477f99-kube-api-access-zljjb\") pod \"87411602-8f4e-4d0e-b93d-fcb560477f99\" (UID: \"87411602-8f4e-4d0e-b93d-fcb560477f99\") " Oct 31 00:40:43.899779 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.899717 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87411602-8f4e-4d0e-b93d-fcb560477f99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87411602-8f4e-4d0e-b93d-fcb560477f99-plugins-config-volume\") pod \"87411602-8f4e-4d0e-b93d-fcb560477f99\" (UID: \"87411602-8f4e-4d0e-b93d-fcb560477f99\") " Oct 31 00:40:43.900009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.899874 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-plugins-config-volume" pod "71c67b9b-d686-40ee-91b8-8a7f8b888ae2" (UID: "71c67b9b-d686-40ee-91b8-8a7f8b888ae2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:43.900091 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.900056 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87411602-8f4e-4d0e-b93d-fcb560477f99-plugins-config-volume" pod "87411602-8f4e-4d0e-b93d-fcb560477f99" (UID: "87411602-8f4e-4d0e-b93d-fcb560477f99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:43.901911 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.901892 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87411602-8f4e-4d0e-b93d-fcb560477f99-kube-api-access-zljjb" pod "87411602-8f4e-4d0e-b93d-fcb560477f99" (UID: "87411602-8f4e-4d0e-b93d-fcb560477f99"). InnerVolumeSpecName "kube-api-access-zljjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:43.901976 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.901906 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-kube-api-access-ggb6p" pod "71c67b9b-d686-40ee-91b8-8a7f8b888ae2" (UID: "71c67b9b-d686-40ee-91b8-8a7f8b888ae2"). InnerVolumeSpecName "kube-api-access-ggb6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:43.941931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.941849 5719 generic.go:356] "Generic (PLEG): container finished" podID="71c67b9b-d686-40ee-91b8-8a7f8b888ae2" containerID="84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da" exitCode=0 Oct 31 00:40:43.941931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.941900 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" event={"ID":"71c67b9b-d686-40ee-91b8-8a7f8b888ae2","Type":"ContainerDied","Data":"84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da"} Oct 31 00:40:43.942131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.941961 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" event={"ID":"71c67b9b-d686-40ee-91b8-8a7f8b888ae2","Type":"ContainerDied","Data":"7fa52423bfd95295f672b69f454fbde3978922bb8a1b46cf297f2eb0a6f5f8ba"} Oct 31 00:40:43.942131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.941910 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" Oct 31 00:40:43.942131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.941995 5719 scope.go:122] "RemoveContainer" containerID="84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da" Oct 31 00:40:43.943614 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.943475 5719 generic.go:356] "Generic (PLEG): container finished" podID="87411602-8f4e-4d0e-b93d-fcb560477f99" containerID="a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb" exitCode=0 Oct 31 00:40:43.943614 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.943519 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" Oct 31 00:40:43.943614 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.943625 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" event={"ID":"87411602-8f4e-4d0e-b93d-fcb560477f99","Type":"ContainerDied","Data":"a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb"} Oct 31 00:40:43.943880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.943658 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" event={"ID":"87411602-8f4e-4d0e-b93d-fcb560477f99","Type":"ContainerDied","Data":"6a89ad59d41f1580ee84f11ee2a6a8322ce9cf6b682186352dba60ef14bd7704"} Oct 31 00:40:43.952926 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.952907 5719 scope.go:122] "RemoveContainer" containerID="84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da" Oct 31 00:40:43.953363 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:40:43.953230 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da\": container with ID starting with 84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da not found: ID does not exist" containerID="84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da" Oct 31 00:40:43.953363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.953251 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da"} err="failed to get container status \"84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da\": rpc error: code = NotFound desc = could not find container \"84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da\": container with ID starting with 84d389b49296dcaf3d5207560993382d0a8b87ba832dd298666f6c32326fc1da not found: ID does not exist" Oct 31 00:40:43.953363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.953268 5719 scope.go:122] "RemoveContainer" containerID="a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb" Oct 31 00:40:43.961487 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.961470 5719 scope.go:122] "RemoveContainer" containerID="a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb" Oct 31 00:40:43.961788 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:40:43.961769 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb\": container with ID starting with a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb not found: ID does not exist" containerID="a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb" Oct 31 00:40:43.961845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:43.961795 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb"} err="failed to get container status \"a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb\": rpc error: code = NotFound desc = could not find container \"a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb\": container with ID starting with a3da2d3e1dc8d04c273df58346191a9cd13b4ae6b1d024fcce4a11fba9b83eeb not found: ID does not exist" Oct 31 00:40:44.001433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.001409 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:44.001558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.001437 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggb6p\" (UniqueName: \"kubernetes.io/projected/71c67b9b-d686-40ee-91b8-8a7f8b888ae2-kube-api-access-ggb6p\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:44.001558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.001452 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zljjb\" (UniqueName: \"kubernetes.io/projected/87411602-8f4e-4d0e-b93d-fcb560477f99-kube-api-access-zljjb\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:44.001558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.001464 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87411602-8f4e-4d0e-b93d-fcb560477f99-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:44.138152 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.138125 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6"] Oct 31 00:40:44.142251 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.142227 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6"] Oct 31 00:40:44.248361 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.248297 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l"] Oct 31 00:40:44.253032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.253003 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l"] Oct 31 00:40:44.602907 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.602869 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c67b9b-d686-40ee-91b8-8a7f8b888ae2" path="/var/lib/kubelet/pods/71c67b9b-d686-40ee-91b8-8a7f8b888ae2/volumes" Oct 31 00:40:44.603352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.603256 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87411602-8f4e-4d0e-b93d-fcb560477f99" path="/var/lib/kubelet/pods/87411602-8f4e-4d0e-b93d-fcb560477f99/volumes" Oct 31 00:40:44.631837 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.631808 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-derive-2b9202f4-5fcb468fc5-8g7c6" podUID="87411602-8f4e-4d0e-b93d-fcb560477f99" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.38:50052\" within 1s: context deadline exceeded" Oct 31 00:40:44.631951 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:44.631812 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-derive-2b9202f4-85ccdf77f7-cm78l" podUID="71c67b9b-d686-40ee-91b8-8a7f8b888ae2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.39:50052\" within 1s: context deadline exceeded" Oct 31 00:40:44.631951 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:40:44.631893 5719 logging.go:55] [core] [Channel #70 SubChannel #71]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.39:50052", ServerName: "10.133.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.39:50052: operation was canceled" Oct 31 00:40:44.631951 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:40:44.631911 5719 logging.go:55] [core] [Channel #68 SubChannel #69]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:50052", ServerName: "10.133.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:50052: operation was canceled" Oct 31 00:40:46.326326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.326226 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" podUID="60bc19a2-3a5a-4597-b027-a445598a64d5" containerName="payload-pre-processing" containerID="cri-o://921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b" gracePeriod=25 Oct 31 00:40:46.570320 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.570298 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:40:46.725529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.725446 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bc19a2-3a5a-4597-b027-a445598a64d5-kube-api-access-rr2ts\" (UniqueName: \"kubernetes.io/projected/60bc19a2-3a5a-4597-b027-a445598a64d5-kube-api-access-rr2ts\") pod \"60bc19a2-3a5a-4597-b027-a445598a64d5\" (UID: \"60bc19a2-3a5a-4597-b027-a445598a64d5\") " Oct 31 00:40:46.725714 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.725578 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60bc19a2-3a5a-4597-b027-a445598a64d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bc19a2-3a5a-4597-b027-a445598a64d5-plugins-config-volume\") pod \"60bc19a2-3a5a-4597-b027-a445598a64d5\" (UID: \"60bc19a2-3a5a-4597-b027-a445598a64d5\") " Oct 31 00:40:46.726198 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.726169 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60bc19a2-3a5a-4597-b027-a445598a64d5-plugins-config-volume" pod "60bc19a2-3a5a-4597-b027-a445598a64d5" (UID: "60bc19a2-3a5a-4597-b027-a445598a64d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:40:46.728399 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.728364 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bc19a2-3a5a-4597-b027-a445598a64d5-kube-api-access-rr2ts" pod "60bc19a2-3a5a-4597-b027-a445598a64d5" (UID: "60bc19a2-3a5a-4597-b027-a445598a64d5"). InnerVolumeSpecName "kube-api-access-rr2ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:40:46.826539 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.826510 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr2ts\" (UniqueName: \"kubernetes.io/projected/60bc19a2-3a5a-4597-b027-a445598a64d5-kube-api-access-rr2ts\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.826653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.826542 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bc19a2-3a5a-4597-b027-a445598a64d5-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:40:46.961861 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.961824 5719 generic.go:356] "Generic (PLEG): container finished" podID="60bc19a2-3a5a-4597-b027-a445598a64d5" containerID="921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b" exitCode=0 Oct 31 00:40:46.962002 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.961871 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" event={"ID":"60bc19a2-3a5a-4597-b027-a445598a64d5","Type":"ContainerDied","Data":"921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b"} Oct 31 00:40:46.962002 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.961876 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" Oct 31 00:40:46.962002 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.961923 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn" event={"ID":"60bc19a2-3a5a-4597-b027-a445598a64d5","Type":"ContainerDied","Data":"d3dfde3cfb26cfd9557123eb4cf9f22664b66b760604ba89a46b58314462e925"} Oct 31 00:40:46.962002 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.961951 5719 scope.go:122] "RemoveContainer" containerID="921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b" Oct 31 00:40:46.971726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.971679 5719 scope.go:122] "RemoveContainer" containerID="921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b" Oct 31 00:40:46.971978 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:40:46.971959 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b\": container with ID starting with 921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b not found: ID does not exist" containerID="921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b" Oct 31 00:40:46.972020 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:46.971987 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b"} err="failed to get container status \"921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b\": rpc error: code = NotFound desc = could not find container \"921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b\": container with ID starting with 921d8e2c833422c70b492f22a9c596f6dfee955c8c186aea910cd7ff5f04341b not found: ID does not exist" Oct 31 00:40:47.122270 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:47.122244 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn"] Oct 31 00:40:47.170010 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:47.169989 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-95fa4445-bc5886d59-xxhhn"] Oct 31 00:40:48.602115 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:48.602081 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60bc19a2-3a5a-4597-b027-a445598a64d5" path="/var/lib/kubelet/pods/60bc19a2-3a5a-4597-b027-a445598a64d5/volumes" Oct 31 00:40:48.951090 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:40:48.951015 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:41:03.008868 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:41:03.008829 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:41:03.271374 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:03.271297 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:41:17.262655 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:17.262624 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:41:18.465577 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.465544 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj"] Oct 31 00:41:18.465984 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.465968 5719 cpu_manager.go:414] "containerMap: removing container" podUID="388f9b15-0c6a-490e-8437-d9aa0f4283ef" containerName="revoke-keys" Oct 31 00:41:18.465984 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.465979 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388f9b15-0c6a-490e-8437-d9aa0f4283ef" containerName="revoke-keys" Oct 31 00:41:18.466062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.465991 5719 cpu_manager.go:414] "containerMap: removing container" podUID="60bc19a2-3a5a-4597-b027-a445598a64d5" containerName="payload-pre-processing" Oct 31 00:41:18.466062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.465996 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bc19a2-3a5a-4597-b027-a445598a64d5" containerName="payload-pre-processing" Oct 31 00:41:18.466062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.466022 5719 cpu_manager.go:414] "containerMap: removing container" podUID="71c67b9b-d686-40ee-91b8-8a7f8b888ae2" containerName="payload-processing" Oct 31 00:41:18.466062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.466027 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c67b9b-d686-40ee-91b8-8a7f8b888ae2" containerName="payload-processing" Oct 31 00:41:18.466062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.466040 5719 cpu_manager.go:414] "containerMap: removing container" podUID="87411602-8f4e-4d0e-b93d-fcb560477f99" containerName="payload-pre-processing" Oct 31 00:41:18.466062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.466045 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87411602-8f4e-4d0e-b93d-fcb560477f99" containerName="payload-pre-processing" Oct 31 00:41:18.466238 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.466101 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c67b9b-d686-40ee-91b8-8a7f8b888ae2" containerName="payload-processing" Oct 31 00:41:18.466238 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.466110 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="87411602-8f4e-4d0e-b93d-fcb560477f99" containerName="payload-pre-processing" Oct 31 00:41:18.466238 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.466116 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="388f9b15-0c6a-490e-8437-d9aa0f4283ef" containerName="revoke-keys" Oct 31 00:41:18.466238 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.466123 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bc19a2-3a5a-4597-b027-a445598a64d5" containerName="payload-pre-processing" Oct 31 00:41:18.469013 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.468991 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.567313 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.567285 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"openshift-service-ca.crt\"" Oct 31 00:41:18.567313 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.567288 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"default-dockercfg-fr2qg\"" Oct 31 00:41:18.567521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.567327 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"simulator-w2-29ece5-kserve-self-signed-certs\"" Oct 31 00:41:18.567521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.567399 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"kube-root-ca.crt\"" Oct 31 00:41:18.568664 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.568647 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj"] Oct 31 00:41:18.588051 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.588023 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-home\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.588189 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.588056 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tmp-dir\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.588189 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.588077 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tls-certs\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.588189 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.588101 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfqr\" (UniqueName: \"kubernetes.io/projected/fa568cd6-7a3e-4f31-8271-781a4f9d4318-kube-api-access-lbfqr\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.588320 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.588260 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-model-cache\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.588320 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.588289 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-dshm\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689202 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689170 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-home\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689202 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689203 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tmp-dir\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689222 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tls-certs\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689237 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfqr\" (UniqueName: \"kubernetes.io/projected/fa568cd6-7a3e-4f31-8271-781a4f9d4318-kube-api-access-lbfqr\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689303 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-model-cache\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689328 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-dshm\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689626 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689566 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-home\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689776 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689631 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tmp-dir\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.689845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.689774 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-model-cache\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.691723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.691678 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-dshm\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.692111 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.692087 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tls-certs\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.748548 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.748490 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfqr\" (UniqueName: \"kubernetes.io/projected/fa568cd6-7a3e-4f31-8271-781a4f9d4318-kube-api-access-lbfqr\") pod \"simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:18.780998 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:18.780965 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:19.152101 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:19.152069 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj"] Oct 31 00:41:19.158109 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:41:19.158081 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa568cd6_7a3e_4f31_8271_781a4f9d4318.slice/crio-97991509ea7a9923721d767d29dcafe168476f153be43999dd68d2bf99c4d364 WatchSource:0}: Error finding container 97991509ea7a9923721d767d29dcafe168476f153be43999dd68d2bf99c4d364: Status 404 returned error can't find the container with id 97991509ea7a9923721d767d29dcafe168476f153be43999dd68d2bf99c4d364 Oct 31 00:41:20.121427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:20.121336 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" event={"ID":"fa568cd6-7a3e-4f31-8271-781a4f9d4318","Type":"ContainerStarted","Data":"97991509ea7a9923721d767d29dcafe168476f153be43999dd68d2bf99c4d364"} Oct 31 00:41:21.128527 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:21.128487 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" event={"ID":"fa568cd6-7a3e-4f31-8271-781a4f9d4318","Type":"ContainerStarted","Data":"d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6"} Oct 31 00:41:21.242631 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:21.242590 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podStartSLOduration=1.738156319 podStartE2EDuration="3.242578609s" totalImagesPullingTime="1.50442229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:21.241776569 +0000 UTC m=+762.777918873" watchObservedRunningTime="2026-10-31 00:41:21.242578609 +0000 UTC m=+762.778720914" Oct 31 00:41:21.781447 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:21.781418 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:24.369076 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:24.369042 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:41:28.781413 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:28.781379 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:28.793911 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:28.793889 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:29.179042 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:29.179016 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:41:33.010156 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:41:33.010125 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:41:39.184134 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:39.184098 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:41:43.995981 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:43.995949 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f"] Oct 31 00:41:44.002965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.002945 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.004265 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.004238 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-dshm\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.004351 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.004303 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tmp-dir\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.004397 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.004354 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84x5\" (UniqueName: \"kubernetes.io/projected/ba6e3e81-a1f3-4266-ae60-ece863ea0416-kube-api-access-c84x5\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.004442 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.004395 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tls-certs\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.004442 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.004417 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-model-cache\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.004442 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.004437 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-home\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.070560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.070530 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ef2572\"/\"openshift-service-ca.crt\"" Oct 31 00:41:44.101304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.101273 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ef2572\"/\"default-dockercfg-59hf4\"" Oct 31 00:41:44.101419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.101325 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ef2572\"/\"kube-root-ca.crt\"" Oct 31 00:41:44.101419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.101345 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ef2572\"/\"simulator-w4-ef2572-kserve-self-signed-certs\"" Oct 31 00:41:44.102917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.102899 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f"] Oct 31 00:41:44.104845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.104827 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tls-certs\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.104889 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.104872 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-model-cache\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.104925 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.104895 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-home\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.104971 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.104937 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-dshm\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.105038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.105000 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tmp-dir\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.105038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.105028 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84x5\" (UniqueName: \"kubernetes.io/projected/ba6e3e81-a1f3-4266-ae60-ece863ea0416-kube-api-access-c84x5\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.105347 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.105315 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-model-cache\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.105458 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.105405 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tmp-dir\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.105458 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.105422 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-home\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.107215 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.107193 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-dshm\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.107298 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.107244 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tls-certs\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.163961 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.163931 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84x5\" (UniqueName: \"kubernetes.io/projected/ba6e3e81-a1f3-4266-ae60-ece863ea0416-kube-api-access-c84x5\") pod \"simulator-w4-ef2572-kserve-5c87dd56c8-56j9f\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.315154 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.315065 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:44.691383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:44.691354 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f"] Oct 31 00:41:44.696988 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:41:44.696963 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba6e3e81_a1f3_4266_ae60_ece863ea0416.slice/crio-3860787eb05ebec6b2cef4cae8ca65960e00c0fca58489518b331c9198220598 WatchSource:0}: Error finding container 3860787eb05ebec6b2cef4cae8ca65960e00c0fca58489518b331c9198220598: Status 404 returned error can't find the container with id 3860787eb05ebec6b2cef4cae8ca65960e00c0fca58489518b331c9198220598 Oct 31 00:41:45.247644 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:45.247611 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" event={"ID":"ba6e3e81-a1f3-4266-ae60-ece863ea0416","Type":"ContainerStarted","Data":"05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2"} Oct 31 00:41:45.248036 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:45.247658 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" event={"ID":"ba6e3e81-a1f3-4266-ae60-ece863ea0416","Type":"ContainerStarted","Data":"3860787eb05ebec6b2cef4cae8ca65960e00c0fca58489518b331c9198220598"} Oct 31 00:41:45.315668 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:45.315646 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:45.360299 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:45.360253 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podStartSLOduration=2.360241856 podStartE2EDuration="2.360241856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:45.359479805 +0000 UTC m=+786.895622109" watchObservedRunningTime="2026-10-31 00:41:45.360241856 +0000 UTC m=+786.896384160" Oct 31 00:41:47.440490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:47.440457 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:41:54.315990 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:54.315955 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:54.328842 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:54.328818 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:41:55.305646 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:41:55.305620 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:42:00.922573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:00.922530 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26"] Oct 31 00:42:00.925352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:00.925330 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.022977 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.022951 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-29ece5\"/\"premium-w2-29ece5-kserve-self-signed-certs\"" Oct 31 00:42:01.024425 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.024405 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26"] Oct 31 00:42:01.049376 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.049345 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-home\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.049500 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.049392 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c759e29-fdad-44b7-a89f-755060c2c483-tls-certs\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.049551 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.049489 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-tmp-dir\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.049551 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.049533 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bw4\" (UniqueName: \"kubernetes.io/projected/3c759e29-fdad-44b7-a89f-755060c2c483-kube-api-access-h9bw4\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.049622 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.049607 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-dshm\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.049752 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.049732 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-model-cache\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.150874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.150840 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-home\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.151041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.150894 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c759e29-fdad-44b7-a89f-755060c2c483-tls-certs\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.151041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.150929 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-tmp-dir\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.151041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.150964 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bw4\" (UniqueName: \"kubernetes.io/projected/3c759e29-fdad-44b7-a89f-755060c2c483-kube-api-access-h9bw4\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.151041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.151000 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-dshm\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.151041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.151027 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-model-cache\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.151347 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.151288 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-home\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.151438 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.151383 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-tmp-dir\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.151541 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.151523 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-model-cache\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.153428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.153403 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-dshm\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.153735 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.153712 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c759e29-fdad-44b7-a89f-755060c2c483-tls-certs\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.209739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.209671 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bw4\" (UniqueName: \"kubernetes.io/projected/3c759e29-fdad-44b7-a89f-755060c2c483-kube-api-access-h9bw4\") pod \"premium-w2-29ece5-kserve-999b8b865-r2q26\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.236621 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.236598 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:01.621272 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:01.621243 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26"] Oct 31 00:42:01.627114 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:42:01.627087 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c759e29_fdad_44b7_a89f_755060c2c483.slice/crio-d92db507175e49d358a3612f70aeeaadc0f1351c2b9e67e484a5177ac144d169 WatchSource:0}: Error finding container d92db507175e49d358a3612f70aeeaadc0f1351c2b9e67e484a5177ac144d169: Status 404 returned error can't find the container with id d92db507175e49d358a3612f70aeeaadc0f1351c2b9e67e484a5177ac144d169 Oct 31 00:42:02.330784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:02.330747 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" event={"ID":"3c759e29-fdad-44b7-a89f-755060c2c483","Type":"ContainerStarted","Data":"3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd"} Oct 31 00:42:02.331163 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:02.330791 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" event={"ID":"3c759e29-fdad-44b7-a89f-755060c2c483","Type":"ContainerStarted","Data":"d92db507175e49d358a3612f70aeeaadc0f1351c2b9e67e484a5177ac144d169"} Oct 31 00:42:02.461536 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:02.461489 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podStartSLOduration=2.461474781 podStartE2EDuration="2.461474781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:02.460368684 +0000 UTC m=+803.996510989" watchObservedRunningTime="2026-10-31 00:42:02.461474781 +0000 UTC m=+803.997617156" Oct 31 00:42:03.009289 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:42:03.009255 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:42:03.237747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:03.237715 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:06.773043 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:06.773009 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:42:11.236958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:11.236924 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:11.249288 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:11.249268 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:11.393640 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:11.393608 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:42:15.841673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.841594 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz"] Oct 31 00:42:15.844733 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.844713 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:15.942898 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.942832 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"kube-root-ca.crt\"" Oct 31 00:42:15.943058 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.943039 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"default-dockercfg-mkgsq\"" Oct 31 00:42:15.943111 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.943087 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"openshift-service-ca.crt\"" Oct 31 00:42:15.943158 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.943121 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b63c24\"/\"simulator-w1-b63c24-kserve-self-signed-certs\"" Oct 31 00:42:15.944518 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.944501 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz"] Oct 31 00:42:15.987530 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.987504 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-home\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:15.987673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.987536 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-dshm\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:15.987673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.987559 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tdf\" (UniqueName: \"kubernetes.io/projected/45a01aef-4f87-4f99-8a1e-b022e707afdb-kube-api-access-r9tdf\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:15.987836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.987684 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a01aef-4f87-4f99-8a1e-b022e707afdb-tls-certs\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:15.987836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.987777 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-model-cache\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:15.987935 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:15.987843 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-tmp-dir\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089263 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-dshm\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089299 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tdf\" (UniqueName: \"kubernetes.io/projected/45a01aef-4f87-4f99-8a1e-b022e707afdb-kube-api-access-r9tdf\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089492 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089357 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a01aef-4f87-4f99-8a1e-b022e707afdb-tls-certs\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089492 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089379 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-model-cache\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089492 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089423 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-tmp-dir\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089499 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-home\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089837 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-model-cache\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089948 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089859 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-tmp-dir\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.089948 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.089920 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-home\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.091830 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.091776 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-dshm\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.092164 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.092134 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a01aef-4f87-4f99-8a1e-b022e707afdb-tls-certs\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.148746 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.148711 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tdf\" (UniqueName: \"kubernetes.io/projected/45a01aef-4f87-4f99-8a1e-b022e707afdb-kube-api-access-r9tdf\") pod \"simulator-w1-b63c24-kserve-66f96bf76d-vwccz\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.177623 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.177596 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:16.547503 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:16.547427 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz"] Oct 31 00:42:16.553498 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:42:16.553477 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45a01aef_4f87_4f99_8a1e_b022e707afdb.slice/crio-b6305957ee83f52bbb6b40aa1c32cc437f3ecc12359286852cda96b45ae4eb66 WatchSource:0}: Error finding container b6305957ee83f52bbb6b40aa1c32cc437f3ecc12359286852cda96b45ae4eb66: Status 404 returned error can't find the container with id b6305957ee83f52bbb6b40aa1c32cc437f3ecc12359286852cda96b45ae4eb66 Oct 31 00:42:17.413147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:17.413111 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" event={"ID":"45a01aef-4f87-4f99-8a1e-b022e707afdb","Type":"ContainerStarted","Data":"0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd"} Oct 31 00:42:17.413516 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:17.413156 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" event={"ID":"45a01aef-4f87-4f99-8a1e-b022e707afdb","Type":"ContainerStarted","Data":"b6305957ee83f52bbb6b40aa1c32cc437f3ecc12359286852cda96b45ae4eb66"} Oct 31 00:42:17.526427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:17.526377 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podStartSLOduration=2.5263645 podStartE2EDuration="2.5263645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:17.525204749 +0000 UTC m=+819.061347055" watchObservedRunningTime="2026-10-31 00:42:17.5263645 +0000 UTC m=+819.062506805" Oct 31 00:42:18.178488 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:18.178445 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:19.870262 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:19.870224 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:42:26.179547 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:26.179508 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:26.192259 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:26.192236 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:26.469584 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:26.469499 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:42:33.009896 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:42:33.009862 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:42:33.359147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.359106 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84"] Oct 31 00:42:33.362017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.362001 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.459422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.459383 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8n6\" (UniqueName: \"kubernetes.io/projected/06cc3e5d-ce95-4db1-be59-c187270251c3-kube-api-access-rb8n6\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.459597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.459443 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-dshm\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.459597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.459539 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-tmp-dir\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.459597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.459590 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06cc3e5d-ce95-4db1-be59-c187270251c3-tls-certs\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.459819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.459730 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-model-cache\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.459819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.459768 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-home\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.459819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.459778 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ef2572\"/\"premium-w4-ef2572-kserve-self-signed-certs\"" Oct 31 00:42:33.461543 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.461526 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84"] Oct 31 00:42:33.561270 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561243 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8n6\" (UniqueName: \"kubernetes.io/projected/06cc3e5d-ce95-4db1-be59-c187270251c3-kube-api-access-rb8n6\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.561448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561281 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-dshm\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.561448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561308 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-tmp-dir\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.561448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561348 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06cc3e5d-ce95-4db1-be59-c187270251c3-tls-certs\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.561448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561377 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-model-cache\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.561448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561406 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-home\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.561812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561788 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-tmp-dir\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.561891 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561812 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-home\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.561891 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.561844 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-model-cache\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.563558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.563539 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-dshm\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.563907 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.563886 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06cc3e5d-ce95-4db1-be59-c187270251c3-tls-certs\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.620219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.620163 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8n6\" (UniqueName: \"kubernetes.io/projected/06cc3e5d-ce95-4db1-be59-c187270251c3-kube-api-access-rb8n6\") pod \"premium-w4-ef2572-kserve-5df897999f-qvk84\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:33.673357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:33.673311 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:34.047895 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:34.047816 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84"] Oct 31 00:42:34.053577 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:42:34.053554 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06cc3e5d_ce95_4db1_be59_c187270251c3.slice/crio-e651c88e7e75d500de4e729954ede2afc069144d5a2b7135f8ace991fe4d86ed WatchSource:0}: Error finding container e651c88e7e75d500de4e729954ede2afc069144d5a2b7135f8ace991fe4d86ed: Status 404 returned error can't find the container with id e651c88e7e75d500de4e729954ede2afc069144d5a2b7135f8ace991fe4d86ed Oct 31 00:42:34.498334 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:34.498299 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" event={"ID":"06cc3e5d-ce95-4db1-be59-c187270251c3","Type":"ContainerStarted","Data":"ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d"} Oct 31 00:42:34.498511 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:34.498343 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" event={"ID":"06cc3e5d-ce95-4db1-be59-c187270251c3","Type":"ContainerStarted","Data":"e651c88e7e75d500de4e729954ede2afc069144d5a2b7135f8ace991fe4d86ed"} Oct 31 00:42:34.611236 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:34.611190 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podStartSLOduration=1.611178306 podStartE2EDuration="1.611178306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:34.610025515 +0000 UTC m=+836.146167821" watchObservedRunningTime="2026-10-31 00:42:34.611178306 +0000 UTC m=+836.147320610" Oct 31 00:42:34.673455 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:34.673415 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:38.957530 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:38.957493 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:42:43.673763 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:43.673731 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:43.685885 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:43.685861 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:44.560128 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:44.560101 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:42:59.396506 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:42:59.396471 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:43:03.010159 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:43:03.010123 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:43:18.381880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.381851 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g"] Oct 31 00:43:18.385429 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.385408 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.393168 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.393145 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g"] Oct 31 00:43:18.435619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.435583 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:43:18.435619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.435589 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:43:18.435619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.435609 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:18.435815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.435677 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:43:18.467021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.466995 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.467116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.467027 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnns\" (UniqueName: \"kubernetes.io/projected/0586fec1-3b38-45af-a13b-e6acb464d2ec-kube-api-access-fhnns\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.467116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.467064 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.568123 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.568096 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.568123 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.568127 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnns\" (UniqueName: \"kubernetes.io/projected/0586fec1-3b38-45af-a13b-e6acb464d2ec-kube-api-access-fhnns\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.568316 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.568162 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.568718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.568675 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.570627 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.570607 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.628049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.628019 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnns\" (UniqueName: \"kubernetes.io/projected/0586fec1-3b38-45af-a13b-e6acb464d2ec-kube-api-access-fhnns\") pod \"maas-api-revoke-sub-e229c3f2-g5k9g\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.699902 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.699840 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:18.891332 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:18.891299 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:43:19.077032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:19.077001 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g"] Oct 31 00:43:19.083100 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:43:19.083060 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0586fec1_3b38_45af_a13b_e6acb464d2ec.slice/crio-8a2781889be2b734673c40912b932b7dd76eeb31c8f86afe9ae7f7bff2e0afcb WatchSource:0}: Error finding container 8a2781889be2b734673c40912b932b7dd76eeb31c8f86afe9ae7f7bff2e0afcb: Status 404 returned error can't find the container with id 8a2781889be2b734673c40912b932b7dd76eeb31c8f86afe9ae7f7bff2e0afcb Oct 31 00:43:19.723409 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:19.723321 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" event={"ID":"0586fec1-3b38-45af-a13b-e6acb464d2ec","Type":"ContainerStarted","Data":"992ec92973f7c11cb1fbef08f9c3f1b1fe3ff814fc25adb48b068dafac475a0b"} Oct 31 00:43:19.723409 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:19.723383 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" event={"ID":"0586fec1-3b38-45af-a13b-e6acb464d2ec","Type":"ContainerStarted","Data":"8a2781889be2b734673c40912b932b7dd76eeb31c8f86afe9ae7f7bff2e0afcb"} Oct 31 00:43:19.835305 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:19.835256 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" podStartSLOduration=1.8352431729999998 podStartE2EDuration="1.835243173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:19.834337259 +0000 UTC m=+881.370479565" watchObservedRunningTime="2026-10-31 00:43:19.835243173 +0000 UTC m=+881.371385477" Oct 31 00:43:20.729919 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:20.729887 5719 generic.go:356] "Generic (PLEG): container finished" podID="0586fec1-3b38-45af-a13b-e6acb464d2ec" containerID="992ec92973f7c11cb1fbef08f9c3f1b1fe3ff814fc25adb48b068dafac475a0b" exitCode=0 Oct 31 00:43:20.730422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:20.729977 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" event={"ID":"0586fec1-3b38-45af-a13b-e6acb464d2ec","Type":"ContainerDied","Data":"992ec92973f7c11cb1fbef08f9c3f1b1fe3ff814fc25adb48b068dafac475a0b"} Oct 31 00:43:21.865707 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:21.865667 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:21.901523 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:21.901492 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-xks-ca\") pod \"0586fec1-3b38-45af-a13b-e6acb464d2ec\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " Oct 31 00:43:21.901661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:21.901608 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-openshift-ca\") pod \"0586fec1-3b38-45af-a13b-e6acb464d2ec\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " Oct 31 00:43:21.901661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:21.901640 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0586fec1-3b38-45af-a13b-e6acb464d2ec-kube-api-access-fhnns\" (UniqueName: \"kubernetes.io/projected/0586fec1-3b38-45af-a13b-e6acb464d2ec-kube-api-access-fhnns\") pod \"0586fec1-3b38-45af-a13b-e6acb464d2ec\" (UID: \"0586fec1-3b38-45af-a13b-e6acb464d2ec\") " Oct 31 00:43:21.901989 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:21.901964 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-openshift-ca" pod "0586fec1-3b38-45af-a13b-e6acb464d2ec" (UID: "0586fec1-3b38-45af-a13b-e6acb464d2ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:21.903666 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:21.903641 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-xks-ca" pod "0586fec1-3b38-45af-a13b-e6acb464d2ec" (UID: "0586fec1-3b38-45af-a13b-e6acb464d2ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:21.903782 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:21.903711 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0586fec1-3b38-45af-a13b-e6acb464d2ec-kube-api-access-fhnns" pod "0586fec1-3b38-45af-a13b-e6acb464d2ec" (UID: "0586fec1-3b38-45af-a13b-e6acb464d2ec"). InnerVolumeSpecName "kube-api-access-fhnns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:22.003190 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:22.003104 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:22.003190 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:22.003135 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhnns\" (UniqueName: \"kubernetes.io/projected/0586fec1-3b38-45af-a13b-e6acb464d2ec-kube-api-access-fhnns\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:22.003190 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:22.003149 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0586fec1-3b38-45af-a13b-e6acb464d2ec-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:22.741097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:22.741066 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" Oct 31 00:43:22.741267 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:22.741121 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g" event={"ID":"0586fec1-3b38-45af-a13b-e6acb464d2ec","Type":"ContainerDied","Data":"8a2781889be2b734673c40912b932b7dd76eeb31c8f86afe9ae7f7bff2e0afcb"} Oct 31 00:43:22.741267 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:22.741164 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a2781889be2b734673c40912b932b7dd76eeb31c8f86afe9ae7f7bff2e0afcb" Oct 31 00:43:26.468979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.468942 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9"] Oct 31 00:43:26.469386 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.469343 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0586fec1-3b38-45af-a13b-e6acb464d2ec" containerName="revoke-keys" Oct 31 00:43:26.469386 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.469354 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0586fec1-3b38-45af-a13b-e6acb464d2ec" containerName="revoke-keys" Oct 31 00:43:26.469461 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.469425 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0586fec1-3b38-45af-a13b-e6acb464d2ec" containerName="revoke-keys" Oct 31 00:43:26.472774 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.472753 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.548560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.548528 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-dshm\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.548718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.548568 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-model-cache\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.548718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.548654 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-home\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.548718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.548681 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76372935-cf71-4ce5-865d-356fc8b8cf76-tls-certs\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.548833 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.548816 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-tmp-dir\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.548866 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.548840 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqb65\" (UniqueName: \"kubernetes.io/projected/76372935-cf71-4ce5-865d-356fc8b8cf76-kube-api-access-zqb65\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.571809 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.571783 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"openshift-service-ca.crt\"" Oct 31 00:43:26.571809 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.571783 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"premium-w3-dd98f0-kserve-self-signed-certs\"" Oct 31 00:43:26.571964 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.571788 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"default-dockercfg-w97p4\"" Oct 31 00:43:26.571964 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.571883 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-dd98f0\"/\"kube-root-ca.crt\"" Oct 31 00:43:26.573498 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.573481 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9"] Oct 31 00:43:26.650058 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650027 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-model-cache\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.650209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650088 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-home\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.650209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650105 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76372935-cf71-4ce5-865d-356fc8b8cf76-tls-certs\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.650324 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650298 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-tmp-dir\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.650387 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650338 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqb65\" (UniqueName: \"kubernetes.io/projected/76372935-cf71-4ce5-865d-356fc8b8cf76-kube-api-access-zqb65\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.650438 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650417 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-dshm\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.650532 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650514 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-model-cache\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.650621 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650559 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-home\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.650621 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.650602 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-tmp-dir\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.652600 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.652581 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-dshm\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.652713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.652682 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76372935-cf71-4ce5-865d-356fc8b8cf76-tls-certs\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.709535 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.709508 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqb65\" (UniqueName: \"kubernetes.io/projected/76372935-cf71-4ce5-865d-356fc8b8cf76-kube-api-access-zqb65\") pod \"premium-w3-dd98f0-kserve-7f8589dcd-8wbt9\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:26.783749 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:26.783646 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:27.171930 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:27.171897 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9"] Oct 31 00:43:27.176837 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:43:27.176807 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76372935_cf71_4ce5_865d_356fc8b8cf76.slice/crio-59ee3126e5176e77748083b9202a517be9ea8e50fcb125cce1053168cf7d27ce WatchSource:0}: Error finding container 59ee3126e5176e77748083b9202a517be9ea8e50fcb125cce1053168cf7d27ce: Status 404 returned error can't find the container with id 59ee3126e5176e77748083b9202a517be9ea8e50fcb125cce1053168cf7d27ce Oct 31 00:43:27.769351 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:27.769317 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" event={"ID":"76372935-cf71-4ce5-865d-356fc8b8cf76","Type":"ContainerStarted","Data":"82998a065ef97ecc5c91c09ec266667d279efbb9d753eec00e2fd35bfb1fb8f1"} Oct 31 00:43:27.769792 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:27.769359 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" event={"ID":"76372935-cf71-4ce5-865d-356fc8b8cf76","Type":"ContainerStarted","Data":"59ee3126e5176e77748083b9202a517be9ea8e50fcb125cce1053168cf7d27ce"} Oct 31 00:43:27.783997 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:27.783976 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:27.882366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:27.882329 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podStartSLOduration=1.882317821 podStartE2EDuration="1.882317821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:27.881536738 +0000 UTC m=+889.417679043" watchObservedRunningTime="2026-10-31 00:43:27.882317821 +0000 UTC m=+889.418460126" Oct 31 00:43:33.009582 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:43:33.009540 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:43:36.337205 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.337180 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s"] Oct 31 00:43:36.340989 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.340968 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.441408 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.441378 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plt45\" (UniqueName: \"kubernetes.io/projected/32a5ee03-d538-46ac-a24b-2a4bba48e80d-kube-api-access-plt45\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.441549 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.441423 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.441605 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.441549 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.442968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.442947 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:36.443089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.442980 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:43:36.443089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.443042 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:43:36.443089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.443087 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:43:36.444641 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.444623 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s"] Oct 31 00:43:36.542518 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.542493 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.542630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.542550 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plt45\" (UniqueName: \"kubernetes.io/projected/32a5ee03-d538-46ac-a24b-2a4bba48e80d-kube-api-access-plt45\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.542630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.542580 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.543196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.543179 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.545053 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.545037 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.601560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.601506 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plt45\" (UniqueName: \"kubernetes.io/projected/32a5ee03-d538-46ac-a24b-2a4bba48e80d-kube-api-access-plt45\") pod \"maas-api-revoke-sub-d0d55191-7sf2s\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.654351 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.654329 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:36.784318 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.784288 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:36.796388 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.796368 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:36.827216 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:36.827196 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:43:37.027016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:37.026954 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s"] Oct 31 00:43:37.033102 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:43:37.033074 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a5ee03_d538_46ac_a24b_2a4bba48e80d.slice/crio-893120ec3d9234c4323e43d582b8d8ddbd84483ee98b6a4dc5ecda3625cd63b7 WatchSource:0}: Error finding container 893120ec3d9234c4323e43d582b8d8ddbd84483ee98b6a4dc5ecda3625cd63b7: Status 404 returned error can't find the container with id 893120ec3d9234c4323e43d582b8d8ddbd84483ee98b6a4dc5ecda3625cd63b7 Oct 31 00:43:37.822442 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:37.822411 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" event={"ID":"32a5ee03-d538-46ac-a24b-2a4bba48e80d","Type":"ContainerStarted","Data":"ce134ebd2dc7447cc3a2b2f7f32cc7fd264fe40c3c71fdbe49a93adb4d1846c9"} Oct 31 00:43:37.822874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:37.822467 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" event={"ID":"32a5ee03-d538-46ac-a24b-2a4bba48e80d","Type":"ContainerStarted","Data":"893120ec3d9234c4323e43d582b8d8ddbd84483ee98b6a4dc5ecda3625cd63b7"} Oct 31 00:43:37.935130 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:37.935087 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" podStartSLOduration=1.9350737869999999 podStartE2EDuration="1.935073787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:37.933367648 +0000 UTC m=+899.469509954" watchObservedRunningTime="2026-10-31 00:43:37.935073787 +0000 UTC m=+899.471216093" Oct 31 00:43:38.561902 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.561871 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j"] Oct 31 00:43:38.567216 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.565670 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.620064 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.620027 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"default-dockercfg-rm4ww\"" Oct 31 00:43:38.660384 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.660357 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2sd\" (UniqueName: \"kubernetes.io/projected/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-kube-api-access-fz2sd\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.660537 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.660403 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tmp-dir\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.660604 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.660543 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-home\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.660604 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.660575 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-dshm\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.660762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.660741 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-model-cache\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.660812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.660791 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tls-certs\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.667180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.667164 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"kube-root-ca.crt\"" Oct 31 00:43:38.667240 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.667163 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"openshift-service-ca.crt\"" Oct 31 00:43:38.667410 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.667397 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"premium-w0-8bec73-kserve-self-signed-certs\"" Oct 31 00:43:38.669096 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.669079 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j"] Oct 31 00:43:38.762135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762100 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tmp-dir\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.762291 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762184 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-home\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.762291 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762274 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-dshm\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.762421 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762404 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-model-cache\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.762462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762434 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tls-certs\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.762539 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762515 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tmp-dir\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.762602 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762528 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2sd\" (UniqueName: \"kubernetes.io/projected/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-kube-api-access-fz2sd\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.762602 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762547 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-home\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.762787 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.762765 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-model-cache\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.764927 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.764907 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tls-certs\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.765057 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.765039 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-dshm\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.821287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.821266 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2sd\" (UniqueName: \"kubernetes.io/projected/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-kube-api-access-fz2sd\") pod \"premium-w0-8bec73-kserve-6cd86f4f94-tfh9j\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:38.830329 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.830298 5719 generic.go:356] "Generic (PLEG): container finished" podID="32a5ee03-d538-46ac-a24b-2a4bba48e80d" containerID="ce134ebd2dc7447cc3a2b2f7f32cc7fd264fe40c3c71fdbe49a93adb4d1846c9" exitCode=0 Oct 31 00:43:38.830642 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.830350 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" event={"ID":"32a5ee03-d538-46ac-a24b-2a4bba48e80d","Type":"ContainerDied","Data":"ce134ebd2dc7447cc3a2b2f7f32cc7fd264fe40c3c71fdbe49a93adb4d1846c9"} Oct 31 00:43:38.884304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:38.884281 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:39.261293 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:39.261218 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j"] Oct 31 00:43:39.266516 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:43:39.266487 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18be2d8e_96aa_4dbd_81b2_2f42df8f7083.slice/crio-6592a90e2b48e5599c48e36bf932c173668dc573c12a66bbdab9c31f4377890c WatchSource:0}: Error finding container 6592a90e2b48e5599c48e36bf932c173668dc573c12a66bbdab9c31f4377890c: Status 404 returned error can't find the container with id 6592a90e2b48e5599c48e36bf932c173668dc573c12a66bbdab9c31f4377890c Oct 31 00:43:39.836874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:39.836829 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" event={"ID":"18be2d8e-96aa-4dbd-81b2-2f42df8f7083","Type":"ContainerStarted","Data":"cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6"} Oct 31 00:43:39.837298 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:39.836891 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" event={"ID":"18be2d8e-96aa-4dbd-81b2-2f42df8f7083","Type":"ContainerStarted","Data":"6592a90e2b48e5599c48e36bf932c173668dc573c12a66bbdab9c31f4377890c"} Oct 31 00:43:39.884924 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:39.884896 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:39.950002 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:39.949955 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podStartSLOduration=1.949940642 podStartE2EDuration="1.949940642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:39.949046048 +0000 UTC m=+901.485188353" watchObservedRunningTime="2026-10-31 00:43:39.949940642 +0000 UTC m=+901.486082947" Oct 31 00:43:39.977375 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:39.977352 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:40.074338 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.074304 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-openshift-ca\") pod \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " Oct 31 00:43:40.074484 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.074352 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a5ee03-d538-46ac-a24b-2a4bba48e80d-kube-api-access-plt45\" (UniqueName: \"kubernetes.io/projected/32a5ee03-d538-46ac-a24b-2a4bba48e80d-kube-api-access-plt45\") pod \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " Oct 31 00:43:40.074565 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.074518 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-xks-ca\") pod \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\" (UID: \"32a5ee03-d538-46ac-a24b-2a4bba48e80d\") " Oct 31 00:43:40.074672 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.074644 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-openshift-ca" pod "32a5ee03-d538-46ac-a24b-2a4bba48e80d" (UID: "32a5ee03-d538-46ac-a24b-2a4bba48e80d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:40.074869 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.074849 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:40.076550 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.076528 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a5ee03-d538-46ac-a24b-2a4bba48e80d-kube-api-access-plt45" pod "32a5ee03-d538-46ac-a24b-2a4bba48e80d" (UID: "32a5ee03-d538-46ac-a24b-2a4bba48e80d"). InnerVolumeSpecName "kube-api-access-plt45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:40.076623 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.076575 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-xks-ca" pod "32a5ee03-d538-46ac-a24b-2a4bba48e80d" (UID: "32a5ee03-d538-46ac-a24b-2a4bba48e80d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:40.175514 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.175457 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plt45\" (UniqueName: \"kubernetes.io/projected/32a5ee03-d538-46ac-a24b-2a4bba48e80d-kube-api-access-plt45\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:40.175514 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.175480 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a5ee03-d538-46ac-a24b-2a4bba48e80d-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:40.846012 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.845977 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" Oct 31 00:43:40.846012 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.845991 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s" event={"ID":"32a5ee03-d538-46ac-a24b-2a4bba48e80d","Type":"ContainerDied","Data":"893120ec3d9234c4323e43d582b8d8ddbd84483ee98b6a4dc5ecda3625cd63b7"} Oct 31 00:43:40.846469 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:40.846071 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="893120ec3d9234c4323e43d582b8d8ddbd84483ee98b6a4dc5ecda3625cd63b7" Oct 31 00:43:42.187236 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.187197 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn"] Oct 31 00:43:42.187713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.187681 5719 cpu_manager.go:414] "containerMap: removing container" podUID="32a5ee03-d538-46ac-a24b-2a4bba48e80d" containerName="revoke-keys" Oct 31 00:43:42.187763 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.187715 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a5ee03-d538-46ac-a24b-2a4bba48e80d" containerName="revoke-keys" Oct 31 00:43:42.187814 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.187805 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a5ee03-d538-46ac-a24b-2a4bba48e80d" containerName="revoke-keys" Oct 31 00:43:42.190997 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.190978 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.245578 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.245556 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn"] Oct 31 00:43:42.290922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.290889 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:43:42.290922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.290908 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:43:42.290922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.290899 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:42.291081 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.290962 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:43:42.294326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.294297 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.294450 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.294412 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.294450 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.294437 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkx68\" (UniqueName: \"kubernetes.io/projected/e71e60b3-25b0-4825-a86f-12648ab061fe-kube-api-access-lkx68\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.395214 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.395192 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.395340 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.395255 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.395340 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.395273 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkx68\" (UniqueName: \"kubernetes.io/projected/e71e60b3-25b0-4825-a86f-12648ab061fe-kube-api-access-lkx68\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.395952 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.395929 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.397590 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.397565 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.453876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.453829 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkx68\" (UniqueName: \"kubernetes.io/projected/e71e60b3-25b0-4825-a86f-12648ab061fe-kube-api-access-lkx68\") pod \"maas-api-revoke-sub-d1015d04-8xpwn\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.503268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.503246 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:42.872882 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:42.872853 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn"] Oct 31 00:43:42.879588 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:43:42.879565 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71e60b3_25b0_4825_a86f_12648ab061fe.slice/crio-8042ada271d305a31c3d917ab00d6300804ef8eb0f859b31c63ee7cadbd033fe WatchSource:0}: Error finding container 8042ada271d305a31c3d917ab00d6300804ef8eb0f859b31c63ee7cadbd033fe: Status 404 returned error can't find the container with id 8042ada271d305a31c3d917ab00d6300804ef8eb0f859b31c63ee7cadbd033fe Oct 31 00:43:43.862029 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:43.862001 5719 generic.go:356] "Generic (PLEG): container finished" podID="e71e60b3-25b0-4825-a86f-12648ab061fe" containerID="3274341d28d240504d6f8d6d198c81fd4ba2579647a67972f3570c1cc3a0a144" exitCode=0 Oct 31 00:43:43.862407 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:43.862103 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" event={"ID":"e71e60b3-25b0-4825-a86f-12648ab061fe","Type":"ContainerDied","Data":"3274341d28d240504d6f8d6d198c81fd4ba2579647a67972f3570c1cc3a0a144"} Oct 31 00:43:43.862407 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:43.862169 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" event={"ID":"e71e60b3-25b0-4825-a86f-12648ab061fe","Type":"ContainerStarted","Data":"8042ada271d305a31c3d917ab00d6300804ef8eb0f859b31c63ee7cadbd033fe"} Oct 31 00:43:45.001865 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.001837 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:45.124662 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.124578 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71e60b3-25b0-4825-a86f-12648ab061fe-kube-api-access-lkx68\" (UniqueName: \"kubernetes.io/projected/e71e60b3-25b0-4825-a86f-12648ab061fe-kube-api-access-lkx68\") pod \"e71e60b3-25b0-4825-a86f-12648ab061fe\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " Oct 31 00:43:45.124662 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.124641 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-xks-ca\") pod \"e71e60b3-25b0-4825-a86f-12648ab061fe\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " Oct 31 00:43:45.124911 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.124855 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-openshift-ca\") pod \"e71e60b3-25b0-4825-a86f-12648ab061fe\" (UID: \"e71e60b3-25b0-4825-a86f-12648ab061fe\") " Oct 31 00:43:45.125462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.125434 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-openshift-ca" pod "e71e60b3-25b0-4825-a86f-12648ab061fe" (UID: "e71e60b3-25b0-4825-a86f-12648ab061fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:45.126353 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.126329 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:45.127237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.127215 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-xks-ca" pod "e71e60b3-25b0-4825-a86f-12648ab061fe" (UID: "e71e60b3-25b0-4825-a86f-12648ab061fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:45.127897 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.127882 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71e60b3-25b0-4825-a86f-12648ab061fe-kube-api-access-lkx68" pod "e71e60b3-25b0-4825-a86f-12648ab061fe" (UID: "e71e60b3-25b0-4825-a86f-12648ab061fe"). InnerVolumeSpecName "kube-api-access-lkx68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:45.227401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.227374 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkx68\" (UniqueName: \"kubernetes.io/projected/e71e60b3-25b0-4825-a86f-12648ab061fe-kube-api-access-lkx68\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:45.227401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.227400 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e71e60b3-25b0-4825-a86f-12648ab061fe-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:45.873562 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.873535 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" Oct 31 00:43:45.873562 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.873541 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn" event={"ID":"e71e60b3-25b0-4825-a86f-12648ab061fe","Type":"ContainerDied","Data":"8042ada271d305a31c3d917ab00d6300804ef8eb0f859b31c63ee7cadbd033fe"} Oct 31 00:43:45.873794 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:45.873585 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8042ada271d305a31c3d917ab00d6300804ef8eb0f859b31c63ee7cadbd033fe" Oct 31 00:43:47.718295 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:47.718261 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:43:48.884620 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:48.884593 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:48.897446 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:48.897425 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:49.896575 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:49.896549 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:43:53.457106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.457075 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s"] Oct 31 00:43:53.457506 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.457497 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e71e60b3-25b0-4825-a86f-12648ab061fe" containerName="revoke-keys" Oct 31 00:43:53.457561 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.457509 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71e60b3-25b0-4825-a86f-12648ab061fe" containerName="revoke-keys" Oct 31 00:43:53.457598 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.457585 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71e60b3-25b0-4825-a86f-12648ab061fe" containerName="revoke-keys" Oct 31 00:43:53.460557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.460530 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.510896 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.510866 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:43:53.510896 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.510888 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:43:53.511043 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.510949 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:43:53.558600 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.558579 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:43:53.561003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.560979 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s"] Oct 31 00:43:53.612015 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.611987 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r525j\" (UniqueName: \"kubernetes.io/projected/acd8b12a-51d9-4205-979a-e22e81fccb68-kube-api-access-r525j\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.612119 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.612106 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.612184 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.612135 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.713850 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.713790 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.713850 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.713826 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.714019 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.713948 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r525j\" (UniqueName: \"kubernetes.io/projected/acd8b12a-51d9-4205-979a-e22e81fccb68-kube-api-access-r525j\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.714583 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.714561 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.716306 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.716290 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:53.774722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:53.774687 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r525j\" (UniqueName: \"kubernetes.io/projected/acd8b12a-51d9-4205-979a-e22e81fccb68-kube-api-access-r525j\") pod \"maas-api-revoke-sub-72dd6170-8td4s\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:54.073741 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:54.073717 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:54.443330 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:54.443261 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s"] Oct 31 00:43:54.448489 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:43:54.448464 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd8b12a_51d9_4205_979a_e22e81fccb68.slice/crio-51e0e904591d4067497fd417b7867720c684361dc8ef51c97c917367a54db10d WatchSource:0}: Error finding container 51e0e904591d4067497fd417b7867720c684361dc8ef51c97c917367a54db10d: Status 404 returned error can't find the container with id 51e0e904591d4067497fd417b7867720c684361dc8ef51c97c917367a54db10d Oct 31 00:43:54.922637 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:54.922601 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" event={"ID":"acd8b12a-51d9-4205-979a-e22e81fccb68","Type":"ContainerStarted","Data":"ed5b159f048a3ab5b9b45325f5f4e2918da2ffd7f0543da44bfcf605eeabb653"} Oct 31 00:43:54.923031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:54.922684 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" event={"ID":"acd8b12a-51d9-4205-979a-e22e81fccb68","Type":"ContainerStarted","Data":"51e0e904591d4067497fd417b7867720c684361dc8ef51c97c917367a54db10d"} Oct 31 00:43:55.034152 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:55.034061 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" podStartSLOduration=2.034050085 podStartE2EDuration="2.034050085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:55.033556824 +0000 UTC m=+916.569699130" watchObservedRunningTime="2026-10-31 00:43:55.034050085 +0000 UTC m=+916.570192390" Oct 31 00:43:55.928995 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:55.928964 5719 generic.go:356] "Generic (PLEG): container finished" podID="acd8b12a-51d9-4205-979a-e22e81fccb68" containerID="ed5b159f048a3ab5b9b45325f5f4e2918da2ffd7f0543da44bfcf605eeabb653" exitCode=0 Oct 31 00:43:55.929413 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:55.929032 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" event={"ID":"acd8b12a-51d9-4205-979a-e22e81fccb68","Type":"ContainerDied","Data":"ed5b159f048a3ab5b9b45325f5f4e2918da2ffd7f0543da44bfcf605eeabb653"} Oct 31 00:43:57.070838 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.070812 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:57.141700 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.141663 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-openshift-ca\") pod \"acd8b12a-51d9-4205-979a-e22e81fccb68\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " Oct 31 00:43:57.141822 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.141801 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-xks-ca\") pod \"acd8b12a-51d9-4205-979a-e22e81fccb68\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " Oct 31 00:43:57.141871 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.141834 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd8b12a-51d9-4205-979a-e22e81fccb68-kube-api-access-r525j\" (UniqueName: \"kubernetes.io/projected/acd8b12a-51d9-4205-979a-e22e81fccb68-kube-api-access-r525j\") pod \"acd8b12a-51d9-4205-979a-e22e81fccb68\" (UID: \"acd8b12a-51d9-4205-979a-e22e81fccb68\") " Oct 31 00:43:57.142070 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.142043 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-openshift-ca" pod "acd8b12a-51d9-4205-979a-e22e81fccb68" (UID: "acd8b12a-51d9-4205-979a-e22e81fccb68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.142172 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.142152 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.143899 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.143879 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-xks-ca" pod "acd8b12a-51d9-4205-979a-e22e81fccb68" (UID: "acd8b12a-51d9-4205-979a-e22e81fccb68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.143996 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.143931 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd8b12a-51d9-4205-979a-e22e81fccb68-kube-api-access-r525j" pod "acd8b12a-51d9-4205-979a-e22e81fccb68" (UID: "acd8b12a-51d9-4205-979a-e22e81fccb68"). InnerVolumeSpecName "kube-api-access-r525j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:57.243673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.243611 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd8b12a-51d9-4205-979a-e22e81fccb68-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.243673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.243634 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r525j\" (UniqueName: \"kubernetes.io/projected/acd8b12a-51d9-4205-979a-e22e81fccb68-kube-api-access-r525j\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.810288 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.810261 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw"] Oct 31 00:43:57.810724 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.810684 5719 cpu_manager.go:414] "containerMap: removing container" podUID="acd8b12a-51d9-4205-979a-e22e81fccb68" containerName="revoke-keys" Oct 31 00:43:57.810773 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.810728 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd8b12a-51d9-4205-979a-e22e81fccb68" containerName="revoke-keys" Oct 31 00:43:57.810842 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.810833 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd8b12a-51d9-4205-979a-e22e81fccb68" containerName="revoke-keys" Oct 31 00:43:57.813993 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.813974 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:57.849114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.849085 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:57.849221 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.849155 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:57.849275 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.849227 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6mq\" (UniqueName: \"kubernetes.io/projected/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-kube-api-access-gz6mq\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:57.869091 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.869070 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw"] Oct 31 00:43:57.940089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.940058 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" Oct 31 00:43:57.940230 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.940095 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s" event={"ID":"acd8b12a-51d9-4205-979a-e22e81fccb68","Type":"ContainerDied","Data":"51e0e904591d4067497fd417b7867720c684361dc8ef51c97c917367a54db10d"} Oct 31 00:43:57.940230 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.940141 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e0e904591d4067497fd417b7867720c684361dc8ef51c97c917367a54db10d" Oct 31 00:43:57.950317 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.950297 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:57.950417 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.950363 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6mq\" (UniqueName: \"kubernetes.io/projected/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-kube-api-access-gz6mq\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:57.950485 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.950470 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:57.951053 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.951031 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:57.952985 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:57.952964 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:58.009529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:58.009501 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6mq\" (UniqueName: \"kubernetes.io/projected/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-kube-api-access-gz6mq\") pod \"maas-api-revoke-sub-66b75af9-k6pzw\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:58.128995 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:58.128971 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:43:58.501740 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:58.501645 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw"] Oct 31 00:43:58.507266 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:43:58.507238 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfe3cf1_23c0_491c_b6af_423989fd6a9b.slice/crio-8776c1e187555a194efe89fb26183e7e61d26bac5fdaff9ae72c36f8bd642797 WatchSource:0}: Error finding container 8776c1e187555a194efe89fb26183e7e61d26bac5fdaff9ae72c36f8bd642797: Status 404 returned error can't find the container with id 8776c1e187555a194efe89fb26183e7e61d26bac5fdaff9ae72c36f8bd642797 Oct 31 00:43:58.946619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:58.946583 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" event={"ID":"5cfe3cf1-23c0-491c-b6af-423989fd6a9b","Type":"ContainerStarted","Data":"747f36227a6750cda2f614795d3e5f7717080dd22244688c172446aa6d611034"} Oct 31 00:43:58.946746 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:58.946644 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" event={"ID":"5cfe3cf1-23c0-491c-b6af-423989fd6a9b","Type":"ContainerStarted","Data":"8776c1e187555a194efe89fb26183e7e61d26bac5fdaff9ae72c36f8bd642797"} Oct 31 00:43:59.058427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:59.058352 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" podStartSLOduration=2.058339786 podStartE2EDuration="2.058339786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:59.057420545 +0000 UTC m=+920.593562854" watchObservedRunningTime="2026-10-31 00:43:59.058339786 +0000 UTC m=+920.594482094" Oct 31 00:43:59.953648 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:59.953613 5719 generic.go:356] "Generic (PLEG): container finished" podID="5cfe3cf1-23c0-491c-b6af-423989fd6a9b" containerID="747f36227a6750cda2f614795d3e5f7717080dd22244688c172446aa6d611034" exitCode=0 Oct 31 00:43:59.954180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:43:59.953661 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" event={"ID":"5cfe3cf1-23c0-491c-b6af-423989fd6a9b","Type":"ContainerDied","Data":"747f36227a6750cda2f614795d3e5f7717080dd22244688c172446aa6d611034"} Oct 31 00:44:01.096955 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.096932 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:44:01.180251 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.180223 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-xks-ca\") pod \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " Oct 31 00:44:01.180395 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.180278 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-kube-api-access-gz6mq\" (UniqueName: \"kubernetes.io/projected/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-kube-api-access-gz6mq\") pod \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " Oct 31 00:44:01.180395 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.180339 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-openshift-ca\") pod \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\" (UID: \"5cfe3cf1-23c0-491c-b6af-423989fd6a9b\") " Oct 31 00:44:01.180718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.180674 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-openshift-ca" pod "5cfe3cf1-23c0-491c-b6af-423989fd6a9b" (UID: "5cfe3cf1-23c0-491c-b6af-423989fd6a9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:01.182387 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.182359 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-kube-api-access-gz6mq" pod "5cfe3cf1-23c0-491c-b6af-423989fd6a9b" (UID: "5cfe3cf1-23c0-491c-b6af-423989fd6a9b"). InnerVolumeSpecName "kube-api-access-gz6mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:01.182473 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.182420 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-xks-ca" pod "5cfe3cf1-23c0-491c-b6af-423989fd6a9b" (UID: "5cfe3cf1-23c0-491c-b6af-423989fd6a9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:01.281966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.281896 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.281966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.281924 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6mq\" (UniqueName: \"kubernetes.io/projected/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-kube-api-access-gz6mq\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.281966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.281939 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe3cf1-23c0-491c-b6af-423989fd6a9b-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:01.964660 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.964631 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" Oct 31 00:44:01.964853 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.964668 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw" event={"ID":"5cfe3cf1-23c0-491c-b6af-423989fd6a9b","Type":"ContainerDied","Data":"8776c1e187555a194efe89fb26183e7e61d26bac5fdaff9ae72c36f8bd642797"} Oct 31 00:44:01.964853 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:01.964734 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8776c1e187555a194efe89fb26183e7e61d26bac5fdaff9ae72c36f8bd642797" Oct 31 00:44:03.008990 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:44:03.008960 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:44:13.722380 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.722346 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n"] Oct 31 00:44:13.722998 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.722977 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5cfe3cf1-23c0-491c-b6af-423989fd6a9b" containerName="revoke-keys" Oct 31 00:44:13.723065 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.723001 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cfe3cf1-23c0-491c-b6af-423989fd6a9b" containerName="revoke-keys" Oct 31 00:44:13.723151 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.723138 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cfe3cf1-23c0-491c-b6af-423989fd6a9b" containerName="revoke-keys" Oct 31 00:44:13.727086 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.727066 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.777712 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.777666 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:13.780020 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.779996 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n"] Oct 31 00:44:13.797742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.797721 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46pzh\" (UniqueName: \"kubernetes.io/projected/ef697917-be6a-441d-9965-82594e1a9a58-kube-api-access-46pzh\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.797853 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.797775 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.797913 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.797871 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.825240 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.825218 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:44:13.825327 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.825244 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:44:13.825327 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.825249 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:44:13.899089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.899067 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46pzh\" (UniqueName: \"kubernetes.io/projected/ef697917-be6a-441d-9965-82594e1a9a58-kube-api-access-46pzh\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.899181 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.899115 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.899181 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.899135 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.899860 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.899802 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.901635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.901613 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:13.958042 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:13.958005 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46pzh\" (UniqueName: \"kubernetes.io/projected/ef697917-be6a-441d-9965-82594e1a9a58-kube-api-access-46pzh\") pod \"maas-api-revoke-sub-872083d7-tzs6n\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:14.039883 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:14.039830 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:14.411353 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:14.411318 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n"] Oct 31 00:44:14.416886 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:44:14.416857 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef697917_be6a_441d_9965_82594e1a9a58.slice/crio-7b468e9473f35128603670b8991c716367415655795cd20f5dd08e63b0f504a2 WatchSource:0}: Error finding container 7b468e9473f35128603670b8991c716367415655795cd20f5dd08e63b0f504a2: Status 404 returned error can't find the container with id 7b468e9473f35128603670b8991c716367415655795cd20f5dd08e63b0f504a2 Oct 31 00:44:15.031064 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:15.030991 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" event={"ID":"ef697917-be6a-441d-9965-82594e1a9a58","Type":"ContainerStarted","Data":"26762eb363061b4d7ee97e19c4daa86286f7d612c5c4ab2311c78d9574e2f5fd"} Oct 31 00:44:15.031064 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:15.031040 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" event={"ID":"ef697917-be6a-441d-9965-82594e1a9a58","Type":"ContainerStarted","Data":"7b468e9473f35128603670b8991c716367415655795cd20f5dd08e63b0f504a2"} Oct 31 00:44:15.143102 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:15.143056 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" podStartSLOduration=2.143044875 podStartE2EDuration="2.143044875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:15.142451765 +0000 UTC m=+936.678594075" watchObservedRunningTime="2026-10-31 00:44:15.143044875 +0000 UTC m=+936.679187222" Oct 31 00:44:16.037810 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:16.037777 5719 generic.go:356] "Generic (PLEG): container finished" podID="ef697917-be6a-441d-9965-82594e1a9a58" containerID="26762eb363061b4d7ee97e19c4daa86286f7d612c5c4ab2311c78d9574e2f5fd" exitCode=0 Oct 31 00:44:16.038196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:16.037861 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" event={"ID":"ef697917-be6a-441d-9965-82594e1a9a58","Type":"ContainerDied","Data":"26762eb363061b4d7ee97e19c4daa86286f7d612c5c4ab2311c78d9574e2f5fd"} Oct 31 00:44:16.975857 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:16.975828 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:44:17.179316 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.179292 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:17.235930 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.235868 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef697917-be6a-441d-9965-82594e1a9a58-kube-api-access-46pzh\" (UniqueName: \"kubernetes.io/projected/ef697917-be6a-441d-9965-82594e1a9a58-kube-api-access-46pzh\") pod \"ef697917-be6a-441d-9965-82594e1a9a58\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " Oct 31 00:44:17.235930 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.235923 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-openshift-ca\") pod \"ef697917-be6a-441d-9965-82594e1a9a58\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " Oct 31 00:44:17.236107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.235985 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-xks-ca\") pod \"ef697917-be6a-441d-9965-82594e1a9a58\" (UID: \"ef697917-be6a-441d-9965-82594e1a9a58\") " Oct 31 00:44:17.236311 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.236287 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-openshift-ca" pod "ef697917-be6a-441d-9965-82594e1a9a58" (UID: "ef697917-be6a-441d-9965-82594e1a9a58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:17.238104 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.238082 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef697917-be6a-441d-9965-82594e1a9a58-kube-api-access-46pzh" pod "ef697917-be6a-441d-9965-82594e1a9a58" (UID: "ef697917-be6a-441d-9965-82594e1a9a58"). InnerVolumeSpecName "kube-api-access-46pzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:17.238219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.238180 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-xks-ca" pod "ef697917-be6a-441d-9965-82594e1a9a58" (UID: "ef697917-be6a-441d-9965-82594e1a9a58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:17.337337 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.337306 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46pzh\" (UniqueName: \"kubernetes.io/projected/ef697917-be6a-441d-9965-82594e1a9a58-kube-api-access-46pzh\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:17.337337 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.337332 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:17.337337 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:17.337342 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef697917-be6a-441d-9965-82594e1a9a58-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:18.050680 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:18.050650 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" Oct 31 00:44:18.050854 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:18.050682 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n" event={"ID":"ef697917-be6a-441d-9965-82594e1a9a58","Type":"ContainerDied","Data":"7b468e9473f35128603670b8991c716367415655795cd20f5dd08e63b0f504a2"} Oct 31 00:44:18.050854 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:18.050748 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b468e9473f35128603670b8991c716367415655795cd20f5dd08e63b0f504a2" Oct 31 00:44:24.507508 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.507475 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72"] Oct 31 00:44:24.507935 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.507922 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ef697917-be6a-441d-9965-82594e1a9a58" containerName="revoke-keys" Oct 31 00:44:24.507935 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.507934 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef697917-be6a-441d-9965-82594e1a9a58" containerName="revoke-keys" Oct 31 00:44:24.508037 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.508026 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef697917-be6a-441d-9965-82594e1a9a58" containerName="revoke-keys" Oct 31 00:44:24.511309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.511290 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.600401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.600372 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.600557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.600445 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.600557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.600534 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6zx\" (UniqueName: \"kubernetes.io/projected/bbbbae9a-9465-478b-9cff-621287e14f77-kube-api-access-zx6zx\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.612069 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.612043 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:44:24.612487 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.612468 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:44:24.612676 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.612662 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:24.613772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.613753 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:44:24.618017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.617996 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72"] Oct 31 00:44:24.701586 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.701558 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.701709 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.701602 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6zx\" (UniqueName: \"kubernetes.io/projected/bbbbae9a-9465-478b-9cff-621287e14f77-kube-api-access-zx6zx\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.701790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.701711 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.702339 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.702317 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.704250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.704226 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.760766 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.760676 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6zx\" (UniqueName: \"kubernetes.io/projected/bbbbae9a-9465-478b-9cff-621287e14f77-kube-api-access-zx6zx\") pod \"maas-api-revoke-sub-56d71785-lzw72\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:24.827811 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:24.827787 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:25.203379 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:25.203351 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72"] Oct 31 00:44:25.208648 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:44:25.208621 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbbae9a_9465_478b_9cff_621287e14f77.slice/crio-48604e4786daacd5acf339a9be6d6c9ea602e91e3b35bd94804868b7a82b4915 WatchSource:0}: Error finding container 48604e4786daacd5acf339a9be6d6c9ea602e91e3b35bd94804868b7a82b4915: Status 404 returned error can't find the container with id 48604e4786daacd5acf339a9be6d6c9ea602e91e3b35bd94804868b7a82b4915 Oct 31 00:44:26.093666 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:26.093632 5719 generic.go:356] "Generic (PLEG): container finished" podID="bbbbae9a-9465-478b-9cff-621287e14f77" containerID="a5a4ccbf2dd017bcc2ba05192e335240d59fb15b229b2b36f1345b306db985f5" exitCode=0 Oct 31 00:44:26.094014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:26.093720 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" event={"ID":"bbbbae9a-9465-478b-9cff-621287e14f77","Type":"ContainerDied","Data":"a5a4ccbf2dd017bcc2ba05192e335240d59fb15b229b2b36f1345b306db985f5"} Oct 31 00:44:26.094014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:26.093771 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" event={"ID":"bbbbae9a-9465-478b-9cff-621287e14f77","Type":"ContainerStarted","Data":"48604e4786daacd5acf339a9be6d6c9ea602e91e3b35bd94804868b7a82b4915"} Oct 31 00:44:27.241708 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.241669 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:27.328558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.328529 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-openshift-ca\") pod \"bbbbae9a-9465-478b-9cff-621287e14f77\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " Oct 31 00:44:27.328677 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.328647 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbbae9a-9465-478b-9cff-621287e14f77-kube-api-access-zx6zx\" (UniqueName: \"kubernetes.io/projected/bbbbae9a-9465-478b-9cff-621287e14f77-kube-api-access-zx6zx\") pod \"bbbbae9a-9465-478b-9cff-621287e14f77\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " Oct 31 00:44:27.328828 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.328812 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-xks-ca\") pod \"bbbbae9a-9465-478b-9cff-621287e14f77\" (UID: \"bbbbae9a-9465-478b-9cff-621287e14f77\") " Oct 31 00:44:27.328925 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.328899 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-openshift-ca" pod "bbbbae9a-9465-478b-9cff-621287e14f77" (UID: "bbbbae9a-9465-478b-9cff-621287e14f77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:27.329121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.329106 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:27.330829 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.330803 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbbae9a-9465-478b-9cff-621287e14f77-kube-api-access-zx6zx" pod "bbbbae9a-9465-478b-9cff-621287e14f77" (UID: "bbbbae9a-9465-478b-9cff-621287e14f77"). InnerVolumeSpecName "kube-api-access-zx6zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:27.330936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.330839 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-xks-ca" pod "bbbbae9a-9465-478b-9cff-621287e14f77" (UID: "bbbbae9a-9465-478b-9cff-621287e14f77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:27.430005 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.429951 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6zx\" (UniqueName: \"kubernetes.io/projected/bbbbae9a-9465-478b-9cff-621287e14f77-kube-api-access-zx6zx\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:27.430005 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:27.429975 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbbbae9a-9465-478b-9cff-621287e14f77-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:28.105404 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:28.105375 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" Oct 31 00:44:28.105584 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:28.105412 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72" event={"ID":"bbbbae9a-9465-478b-9cff-621287e14f77","Type":"ContainerDied","Data":"48604e4786daacd5acf339a9be6d6c9ea602e91e3b35bd94804868b7a82b4915"} Oct 31 00:44:28.105584 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:28.105469 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48604e4786daacd5acf339a9be6d6c9ea602e91e3b35bd94804868b7a82b4915" Oct 31 00:44:29.070504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.070463 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7"] Oct 31 00:44:29.071041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.070898 5719 cpu_manager.go:414] "containerMap: removing container" podUID="bbbbae9a-9465-478b-9cff-621287e14f77" containerName="revoke-keys" Oct 31 00:44:29.071041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.070910 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbbae9a-9465-478b-9cff-621287e14f77" containerName="revoke-keys" Oct 31 00:44:29.071041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.070993 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbbae9a-9465-478b-9cff-621287e14f77" containerName="revoke-keys" Oct 31 00:44:29.074183 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.074160 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.124930 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.124905 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ef2572-openshift-default-dockercfg-fvb72\"" Oct 31 00:44:29.175778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.175752 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7"] Oct 31 00:44:29.248049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248021 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.248247 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248063 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.248247 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248083 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.248247 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248101 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.248448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248283 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9973703c-628f-40b6-9aae-df327ba8ad65-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.248448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248376 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.248448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248395 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9973703c-628f-40b6-9aae-df327ba8ad65-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.248448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248417 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.248448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.248436 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcvbk\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-kube-api-access-kcvbk\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350045 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9973703c-628f-40b6-9aae-df327ba8ad65-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350112 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350130 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9973703c-628f-40b6-9aae-df327ba8ad65-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350159 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350180 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcvbk\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-kube-api-access-kcvbk\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350208 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350246 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350275 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350299 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350822 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350785 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350822 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350814 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350898 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.350936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.350913 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.351152 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.351128 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9973703c-628f-40b6-9aae-df327ba8ad65-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.352656 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.352630 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.352918 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.352903 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9973703c-628f-40b6-9aae-df327ba8ad65-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.408593 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.408561 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.408790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.408771 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcvbk\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-kube-api-access-kcvbk\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:29.687528 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:29.687462 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:30.116464 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:30.116425 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7"] Oct 31 00:44:30.125602 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:30.125564 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:30.125747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:30.125652 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:30.125747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:30.125717 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:31.127291 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:31.127250 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" event={"ID":"9973703c-628f-40b6-9aae-df327ba8ad65","Type":"ContainerStarted","Data":"af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c"} Oct 31 00:44:31.127781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:31.127315 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" event={"ID":"9973703c-628f-40b6-9aae-df327ba8ad65","Type":"ContainerStarted","Data":"e26ecfabc54b0753e3714a9d745245f1b5f8ddc2ccb2dac2cd26677613a546d3"} Oct 31 00:44:31.340187 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:31.340129 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" podStartSLOduration=2.34011489 podStartE2EDuration="2.34011489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:31.291066784 +0000 UTC m=+952.827209089" watchObservedRunningTime="2026-10-31 00:44:31.34011489 +0000 UTC m=+952.876257199" Oct 31 00:44:31.687885 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:31.687849 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:31.689427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:31.689399 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:44:31.689556 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:31.689445 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:44:32.688306 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:32.688274 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:44:32.688713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:32.688331 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:44:33.022767 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.015908 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c"] Oct 31 00:44:33.030752 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.030726 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.032318 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:44:33.032281 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:44:33.070565 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.070539 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c"] Oct 31 00:44:33.122987 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.122957 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:44:33.123922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.123836 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:44:33.123922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.123911 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:44:33.124258 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.124226 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:44:33.192622 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.192590 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsv2h\" (UniqueName: \"kubernetes.io/projected/90de1006-9d07-4a4c-b294-3fc1baadcde6-kube-api-access-nsv2h\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.192786 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.192631 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.192786 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.192664 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.293869 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.293787 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsv2h\" (UniqueName: \"kubernetes.io/projected/90de1006-9d07-4a4c-b294-3fc1baadcde6-kube-api-access-nsv2h\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.293869 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.293822 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.294083 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.293946 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.294511 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.294489 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.296653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.296622 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.353571 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.353542 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsv2h\" (UniqueName: \"kubernetes.io/projected/90de1006-9d07-4a4c-b294-3fc1baadcde6-kube-api-access-nsv2h\") pod \"maas-api-revoke-sub-9f793c98-mpx9c\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.643772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.643742 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:33.688126 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.688091 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:44:33.688259 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:33.688164 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:44:34.026826 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:44:34.026791 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90de1006_9d07_4a4c_b294_3fc1baadcde6.slice/crio-d9fbd12e1313f66d3cb64a9124847affd66d9e1f833e9437ec141d618c689bc6 WatchSource:0}: Error finding container d9fbd12e1313f66d3cb64a9124847affd66d9e1f833e9437ec141d618c689bc6: Status 404 returned error can't find the container with id d9fbd12e1313f66d3cb64a9124847affd66d9e1f833e9437ec141d618c689bc6 Oct 31 00:44:34.028210 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:34.028183 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c"] Oct 31 00:44:34.146561 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:34.146526 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" event={"ID":"90de1006-9d07-4a4c-b294-3fc1baadcde6","Type":"ContainerStarted","Data":"d9fbd12e1313f66d3cb64a9124847affd66d9e1f833e9437ec141d618c689bc6"} Oct 31 00:44:34.688740 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:34.688647 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:44:34.688740 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:34.688725 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:44:35.153417 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:35.153385 5719 generic.go:356] "Generic (PLEG): container finished" podID="90de1006-9d07-4a4c-b294-3fc1baadcde6" containerID="38fe59bc24acd08642f52fb1fc23f59ef3a8ee3a427be833d7734cc4c2be3ab1" exitCode=0 Oct 31 00:44:35.153789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:35.153432 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" event={"ID":"90de1006-9d07-4a4c-b294-3fc1baadcde6","Type":"ContainerDied","Data":"38fe59bc24acd08642f52fb1fc23f59ef3a8ee3a427be833d7734cc4c2be3ab1"} Oct 31 00:44:35.688209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:35.688176 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:44:35.688387 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:35.688226 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:44:36.341529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.341503 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:36.451837 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.451755 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90de1006-9d07-4a4c-b294-3fc1baadcde6-kube-api-access-nsv2h\" (UniqueName: \"kubernetes.io/projected/90de1006-9d07-4a4c-b294-3fc1baadcde6-kube-api-access-nsv2h\") pod \"90de1006-9d07-4a4c-b294-3fc1baadcde6\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " Oct 31 00:44:36.451837 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.451800 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-openshift-ca\") pod \"90de1006-9d07-4a4c-b294-3fc1baadcde6\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " Oct 31 00:44:36.452037 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.451875 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-xks-ca\") pod \"90de1006-9d07-4a4c-b294-3fc1baadcde6\" (UID: \"90de1006-9d07-4a4c-b294-3fc1baadcde6\") " Oct 31 00:44:36.452281 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.452230 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-openshift-ca" pod "90de1006-9d07-4a4c-b294-3fc1baadcde6" (UID: "90de1006-9d07-4a4c-b294-3fc1baadcde6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:36.452410 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.452390 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:36.454542 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.454512 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-xks-ca" pod "90de1006-9d07-4a4c-b294-3fc1baadcde6" (UID: "90de1006-9d07-4a4c-b294-3fc1baadcde6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:36.454647 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.454610 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90de1006-9d07-4a4c-b294-3fc1baadcde6-kube-api-access-nsv2h" pod "90de1006-9d07-4a4c-b294-3fc1baadcde6" (UID: "90de1006-9d07-4a4c-b294-3fc1baadcde6"). InnerVolumeSpecName "kube-api-access-nsv2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:36.553658 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.553624 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsv2h\" (UniqueName: \"kubernetes.io/projected/90de1006-9d07-4a4c-b294-3fc1baadcde6-kube-api-access-nsv2h\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:36.553658 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.553660 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90de1006-9d07-4a4c-b294-3fc1baadcde6-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:36.688479 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.688449 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 31 00:44:36.688610 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:36.688499 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 31 00:44:37.166827 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.166793 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" Oct 31 00:44:37.166827 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.166824 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c" event={"ID":"90de1006-9d07-4a4c-b294-3fc1baadcde6","Type":"ContainerDied","Data":"d9fbd12e1313f66d3cb64a9124847affd66d9e1f833e9437ec141d618c689bc6"} Oct 31 00:44:37.167094 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.166867 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9fbd12e1313f66d3cb64a9124847affd66d9e1f833e9437ec141d618c689bc6" Oct 31 00:44:37.370883 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.370850 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp"] Oct 31 00:44:37.371510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.371488 5719 cpu_manager.go:414] "containerMap: removing container" podUID="90de1006-9d07-4a4c-b294-3fc1baadcde6" containerName="revoke-keys" Oct 31 00:44:37.371551 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.371515 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90de1006-9d07-4a4c-b294-3fc1baadcde6" containerName="revoke-keys" Oct 31 00:44:37.371726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.371712 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="90de1006-9d07-4a4c-b294-3fc1baadcde6" containerName="revoke-keys" Oct 31 00:44:37.375577 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.375555 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.462620 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.462540 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxx4n\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-kube-api-access-nxx4n\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.462620 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.462584 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d97e3-7f91-4460-b598-b659e262afb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.462859 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.462766 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.462859 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.462788 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.462859 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.462823 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.463035 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.462924 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d97e3-7f91-4460-b598-b659e262afb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.463035 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.462962 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.463147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.463122 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.463205 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.463162 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.564736 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.564551 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.564736 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.564597 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.564736 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.564639 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.564736 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.564678 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d97e3-7f91-4460-b598-b659e262afb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.565097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.564756 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.565097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.564930 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.565097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.564952 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.565097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.564984 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxx4n\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-kube-api-access-nxx4n\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.565097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.565005 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d97e3-7f91-4460-b598-b659e262afb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.565352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.565265 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.565538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.565510 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.565677 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.565652 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d97e3-7f91-4460-b598-b659e262afb3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.566079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.565815 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.566230 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.566171 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.568760 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.568732 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.570488 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.570449 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d97e3-7f91-4460-b598-b659e262afb3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.570681 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.570654 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp"] Oct 31 00:44:37.624566 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.624535 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxx4n\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-kube-api-access-nxx4n\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.626736 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.626715 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kj5vp\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.689140 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.689116 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:37.691941 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.691922 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:37.692384 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.692367 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:37.693427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:37.693128 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 00:44:38.114261 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:38.114226 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp"] Oct 31 00:44:38.118888 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:44:38.118864 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38d97e3_7f91_4460_b598_b659e262afb3.slice/crio-db746780c5bbdebae53967a89a02e6975aebe4980d48f97f55f5dd1c9c8e602f WatchSource:0}: Error finding container db746780c5bbdebae53967a89a02e6975aebe4980d48f97f55f5dd1c9c8e602f: Status 404 returned error can't find the container with id db746780c5bbdebae53967a89a02e6975aebe4980d48f97f55f5dd1c9c8e602f Oct 31 00:44:38.121223 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:38.121193 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:38.121297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:38.121260 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:38.121297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:38.121291 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:38.174665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:38.174638 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" event={"ID":"f38d97e3-7f91-4460-b598-b659e262afb3","Type":"ContainerStarted","Data":"db746780c5bbdebae53967a89a02e6975aebe4980d48f97f55f5dd1c9c8e602f"} Oct 31 00:44:39.182496 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:39.182457 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" event={"ID":"f38d97e3-7f91-4460-b598-b659e262afb3","Type":"ContainerStarted","Data":"e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6"} Oct 31 00:44:39.316717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:39.310296 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podStartSLOduration=2.310277283 podStartE2EDuration="2.310277283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:39.30821336 +0000 UTC m=+960.844355666" watchObservedRunningTime="2026-10-31 00:44:39.310277283 +0000 UTC m=+960.846419589" Oct 31 00:44:39.689459 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:39.689428 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:39.690681 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:39.690651 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kj5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:44:39.690844 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:39.690725 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:44:40.690057 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:40.690027 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kj5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:44:40.690446 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:40.690070 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:44:41.689815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:41.689782 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kj5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:44:41.690011 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:41.689847 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:44:42.689609 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:42.689565 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kj5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:44:42.690116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:42.689634 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:44:43.423716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:43.423438 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:44:43.689836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:43.689750 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kj5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:44:43.690271 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:43.689834 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:44:44.690196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:44.690163 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kj5vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 00:44:44.690682 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:44.690215 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 00:44:46.573598 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:46.573554 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:46.574196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:46.574014 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:46.574956 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:46.574936 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 00:44:52.472105 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.472070 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p"] Oct 31 00:44:52.476110 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.476087 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.574226 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.574192 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p"] Oct 31 00:44:52.633236 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633205 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.633377 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633250 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.633377 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633272 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.633377 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633356 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.633473 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633404 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxrz\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-kube-api-access-xmxrz\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.633504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633476 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f5caf1-12b8-4298-a800-5cb5811455ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.633504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633499 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.633583 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633519 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.633583 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.633541 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734136 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734070 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734136 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734103 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734151 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734177 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734194 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734216 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734250 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxrz\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-kube-api-access-xmxrz\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734297 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f5caf1-12b8-4298-a800-5cb5811455ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734606 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734327 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734606 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734510 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734679 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734652 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734792 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734682 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.734988 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.734964 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.735131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.735113 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f5caf1-12b8-4298-a800-5cb5811455ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.736743 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.736718 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.736853 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.736823 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.793007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.792988 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:52.793252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:52.793236 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxrz\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-kube-api-access-xmxrz\") pod \"maas-default-gateway-openshift-default-6cc458d887-sf86p\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:53.088902 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:53.088871 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:53.516893 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:53.516822 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p"] Oct 31 00:44:53.522457 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:44:53.522433 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f5caf1_12b8_4298_a800_5cb5811455ab.slice/crio-3e1af41b804bfe913a8f4cdaf5b2334e68e7c2a2d744aa54cd7d22deb191f4e9 WatchSource:0}: Error finding container 3e1af41b804bfe913a8f4cdaf5b2334e68e7c2a2d744aa54cd7d22deb191f4e9: Status 404 returned error can't find the container with id 3e1af41b804bfe913a8f4cdaf5b2334e68e7c2a2d744aa54cd7d22deb191f4e9 Oct 31 00:44:53.524948 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:53.524919 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:53.525014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:53.524983 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:53.525047 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:53.525013 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:44:54.333677 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:54.333637 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" event={"ID":"22f5caf1-12b8-4298-a800-5cb5811455ab","Type":"ContainerStarted","Data":"a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70"} Oct 31 00:44:54.333879 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:54.333739 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" event={"ID":"22f5caf1-12b8-4298-a800-5cb5811455ab","Type":"ContainerStarted","Data":"3e1af41b804bfe913a8f4cdaf5b2334e68e7c2a2d744aa54cd7d22deb191f4e9"} Oct 31 00:44:54.503717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:54.499799 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podStartSLOduration=2.499780929 podStartE2EDuration="2.499780929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:54.497478977 +0000 UTC m=+976.033621287" watchObservedRunningTime="2026-10-31 00:44:54.499780929 +0000 UTC m=+976.035923234" Oct 31 00:44:55.089882 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:55.089844 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:44:55.091529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:55.091504 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:44:55.091645 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:55.091567 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:44:56.089608 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:56.089563 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:44:56.089814 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:56.089643 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:44:57.090128 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:57.089900 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:44:57.090128 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:57.089992 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:44:58.089346 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:58.089310 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:44:58.089522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:58.089362 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:44:59.089382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:59.089349 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:44:59.089880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:44:59.089405 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:00.090073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.090035 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:00.090558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.090098 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:00.196474 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.196443 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd"] Oct 31 00:45:00.200554 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.200533 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb"] Oct 31 00:45:00.200720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.200680 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:00.203861 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.203843 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk"] Oct 31 00:45:00.203969 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.203953 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:00.207196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.207174 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:00.238179 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.238106 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8901e-25ef-4430-b35c-17556e95d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb\" (UID: \"5fc8901e-25ef-4430-b35c-17556e95d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:00.238328 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.238305 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237897e0-a086-41fd-b910-a82883ace527-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd\" (UID: \"237897e0-a086-41fd-b910-a82883ace527\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:00.238373 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.238343 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5br5j\" (UniqueName: \"kubernetes.io/projected/237897e0-a086-41fd-b910-a82883ace527-kube-api-access-5br5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd\" (UID: \"237897e0-a086-41fd-b910-a82883ace527\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:00.238449 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.238434 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2kf8\" (UniqueName: \"kubernetes.io/projected/5fc8901e-25ef-4430-b35c-17556e95d517-kube-api-access-f2kf8\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb\" (UID: \"5fc8901e-25ef-4430-b35c-17556e95d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:00.238487 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.238465 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-5pqwk\" (UID: \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:00.238535 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.238518 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p62nn\" (UniqueName: \"kubernetes.io/projected/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-kube-api-access-p62nn\") pod \"maas-api-key-cleanup-29890125-5pqwk\" (UID: \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:00.243906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.243888 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9"] Oct 31 00:45:00.247559 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.247543 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:00.251897 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.251874 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:45:00.251995 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.251885 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:45:00.251995 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.251989 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:45:00.311365 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.311338 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd"] Oct 31 00:45:00.315795 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.315749 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk"] Oct 31 00:45:00.318767 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.318744 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb"] Oct 31 00:45:00.339999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.339971 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8901e-25ef-4430-b35c-17556e95d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb\" (UID: \"5fc8901e-25ef-4430-b35c-17556e95d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:00.340183 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340067 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237897e0-a086-41fd-b910-a82883ace527-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd\" (UID: \"237897e0-a086-41fd-b910-a82883ace527\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:00.340183 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340095 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5br5j\" (UniqueName: \"kubernetes.io/projected/237897e0-a086-41fd-b910-a82883ace527-kube-api-access-5br5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd\" (UID: \"237897e0-a086-41fd-b910-a82883ace527\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:00.340183 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340144 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2kf8\" (UniqueName: \"kubernetes.io/projected/5fc8901e-25ef-4430-b35c-17556e95d517-kube-api-access-f2kf8\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb\" (UID: \"5fc8901e-25ef-4430-b35c-17556e95d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:00.340183 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340174 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-5pqwk\" (UID: \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:00.340470 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340224 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p62nn\" (UniqueName: \"kubernetes.io/projected/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-kube-api-access-p62nn\") pod \"maas-api-key-cleanup-29890125-5pqwk\" (UID: \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:00.340470 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340275 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9\" (UID: \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:00.340470 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340320 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d9kj\" (UniqueName: \"kubernetes.io/projected/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-kube-api-access-8d9kj\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9\" (UID: \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:00.340772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340752 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8901e-25ef-4430-b35c-17556e95d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb\" (UID: \"5fc8901e-25ef-4430-b35c-17556e95d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:00.340926 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.340853 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237897e0-a086-41fd-b910-a82883ace527-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd\" (UID: \"237897e0-a086-41fd-b910-a82883ace527\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:00.341431 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.341409 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890125-5pqwk\" (UID: \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:00.353636 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.352819 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9"] Oct 31 00:45:00.400304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.400278 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5br5j\" (UniqueName: \"kubernetes.io/projected/237897e0-a086-41fd-b910-a82883ace527-kube-api-access-5br5j\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd\" (UID: \"237897e0-a086-41fd-b910-a82883ace527\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:00.401774 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.401729 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p62nn\" (UniqueName: \"kubernetes.io/projected/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-kube-api-access-p62nn\") pod \"maas-api-key-cleanup-29890125-5pqwk\" (UID: \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:00.401883 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.401803 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2kf8\" (UniqueName: \"kubernetes.io/projected/5fc8901e-25ef-4430-b35c-17556e95d517-kube-api-access-f2kf8\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb\" (UID: \"5fc8901e-25ef-4430-b35c-17556e95d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:00.441218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.441191 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9\" (UID: \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:00.441326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.441230 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d9kj\" (UniqueName: \"kubernetes.io/projected/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-kube-api-access-8d9kj\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9\" (UID: \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:00.441860 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.441842 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9\" (UID: \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:00.501902 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.501873 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d9kj\" (UniqueName: \"kubernetes.io/projected/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-kube-api-access-8d9kj\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9\" (UID: \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:00.527343 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.527318 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:00.538521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.538505 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:00.544585 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.544563 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:00.560326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.560300 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:00.921936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.921864 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd"] Oct 31 00:45:00.927469 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:45:00.927444 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237897e0_a086_41fd_b910_a82883ace527.slice/crio-3845f818b60040500ee41d9314f332ce6db4a41511e6578c9a9d29a4653ec1be WatchSource:0}: Error finding container 3845f818b60040500ee41d9314f332ce6db4a41511e6578c9a9d29a4653ec1be: Status 404 returned error can't find the container with id 3845f818b60040500ee41d9314f332ce6db4a41511e6578c9a9d29a4653ec1be Oct 31 00:45:00.973802 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.973722 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb"] Oct 31 00:45:00.976899 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.976875 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9"] Oct 31 00:45:00.979776 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:00.979749 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk"] Oct 31 00:45:00.980556 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:45:00.980530 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc8901e_25ef_4430_b35c_17556e95d517.slice/crio-10779fa343c80c9e2474d088306bd81d4762d6a87842cbf059737e8314cc9b97 WatchSource:0}: Error finding container 10779fa343c80c9e2474d088306bd81d4762d6a87842cbf059737e8314cc9b97: Status 404 returned error can't find the container with id 10779fa343c80c9e2474d088306bd81d4762d6a87842cbf059737e8314cc9b97 Oct 31 00:45:00.981377 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:45:00.981359 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3d9e07_f23d_4d5c_95a8_d5f434b0d41b.slice/crio-f7edcad35c9dee7f2930a73a2e15eed7e927ed16cf7ca29bc4b070b22a71dd81 WatchSource:0}: Error finding container f7edcad35c9dee7f2930a73a2e15eed7e927ed16cf7ca29bc4b070b22a71dd81: Status 404 returned error can't find the container with id f7edcad35c9dee7f2930a73a2e15eed7e927ed16cf7ca29bc4b070b22a71dd81 Oct 31 00:45:00.982813 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:45:00.982782 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea5a75a_316c_4cbe_ada8_2a2c9495e10c.slice/crio-461b7e936b3e7b324075eab0f23790656ecf39a8ad2af72b051e5856fe57a6a8 WatchSource:0}: Error finding container 461b7e936b3e7b324075eab0f23790656ecf39a8ad2af72b051e5856fe57a6a8: Status 404 returned error can't find the container with id 461b7e936b3e7b324075eab0f23790656ecf39a8ad2af72b051e5856fe57a6a8 Oct 31 00:45:01.089412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:01.089382 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 00:45:01.089544 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:01.089428 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 00:45:01.398758 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:01.398728 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" event={"ID":"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c","Type":"ContainerStarted","Data":"461b7e936b3e7b324075eab0f23790656ecf39a8ad2af72b051e5856fe57a6a8"} Oct 31 00:45:01.406131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:01.406100 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" event={"ID":"5fc8901e-25ef-4430-b35c-17556e95d517","Type":"ContainerStarted","Data":"10779fa343c80c9e2474d088306bd81d4762d6a87842cbf059737e8314cc9b97"} Oct 31 00:45:01.408083 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:01.408047 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" event={"ID":"237897e0-a086-41fd-b910-a82883ace527","Type":"ContainerStarted","Data":"a938e11a496319c0e6dc8bb81af8921665778c6059eadec77ddbc7d313097cfb"} Oct 31 00:45:01.408232 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:01.408092 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" event={"ID":"237897e0-a086-41fd-b910-a82883ace527","Type":"ContainerStarted","Data":"3845f818b60040500ee41d9314f332ce6db4a41511e6578c9a9d29a4653ec1be"} Oct 31 00:45:01.410232 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:01.410206 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" event={"ID":"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b","Type":"ContainerStarted","Data":"f7edcad35c9dee7f2930a73a2e15eed7e927ed16cf7ca29bc4b070b22a71dd81"} Oct 31 00:45:01.519425 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:01.519355 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" podStartSLOduration=1.519341484 podStartE2EDuration="1.519341484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.518795012 +0000 UTC m=+983.054937317" watchObservedRunningTime="2026-10-31 00:45:01.519341484 +0000 UTC m=+983.055483791" Oct 31 00:45:02.423222 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.423187 5719 generic.go:356] "Generic (PLEG): container finished" podID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerID="1329c1743bb6fe172aa491dcd9979ac3552bf9185310ff3ea2856bc246373770" exitCode=60 Oct 31 00:45:02.423781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.423285 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" event={"ID":"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b","Type":"ContainerDied","Data":"1329c1743bb6fe172aa491dcd9979ac3552bf9185310ff3ea2856bc246373770"} Oct 31 00:45:02.423781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.423483 5719 scope.go:122] "RemoveContainer" containerID="1329c1743bb6fe172aa491dcd9979ac3552bf9185310ff3ea2856bc246373770" Oct 31 00:45:02.425211 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.425185 5719 generic.go:356] "Generic (PLEG): container finished" podID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerID="7ecb2f0f69d6bf0eda0c7ef5af07aa1f8750ed11367f60bc0e794ac2da86d291" exitCode=60 Oct 31 00:45:02.425303 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.425252 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" event={"ID":"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c","Type":"ContainerDied","Data":"7ecb2f0f69d6bf0eda0c7ef5af07aa1f8750ed11367f60bc0e794ac2da86d291"} Oct 31 00:45:02.425432 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.425419 5719 scope.go:122] "RemoveContainer" containerID="7ecb2f0f69d6bf0eda0c7ef5af07aa1f8750ed11367f60bc0e794ac2da86d291" Oct 31 00:45:02.426966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.426943 5719 generic.go:356] "Generic (PLEG): container finished" podID="5fc8901e-25ef-4430-b35c-17556e95d517" containerID="a88696fab083a2cc4516667262155de5ec9d89bdbb70082e9c4e062f7eed162e" exitCode=60 Oct 31 00:45:02.427077 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.427024 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" event={"ID":"5fc8901e-25ef-4430-b35c-17556e95d517","Type":"ContainerDied","Data":"a88696fab083a2cc4516667262155de5ec9d89bdbb70082e9c4e062f7eed162e"} Oct 31 00:45:02.427215 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.427196 5719 scope.go:122] "RemoveContainer" containerID="a88696fab083a2cc4516667262155de5ec9d89bdbb70082e9c4e062f7eed162e" Oct 31 00:45:02.428848 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.428828 5719 generic.go:356] "Generic (PLEG): container finished" podID="237897e0-a086-41fd-b910-a82883ace527" containerID="a938e11a496319c0e6dc8bb81af8921665778c6059eadec77ddbc7d313097cfb" exitCode=60 Oct 31 00:45:02.428956 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.428850 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" event={"ID":"237897e0-a086-41fd-b910-a82883ace527","Type":"ContainerDied","Data":"a938e11a496319c0e6dc8bb81af8921665778c6059eadec77ddbc7d313097cfb"} Oct 31 00:45:02.429085 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:02.429067 5719 scope.go:122] "RemoveContainer" containerID="a938e11a496319c0e6dc8bb81af8921665778c6059eadec77ddbc7d313097cfb" Oct 31 00:45:03.011665 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:03.011609 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:45:03.131841 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.131799 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:45:03.132036 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.131878 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:03.437767 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.437736 5719 generic.go:356] "Generic (PLEG): container finished" podID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerID="0dac6e9b79d5e126bdd77f6698f33ab0cf88636ab4b1d2d6413b517ce5f960e6" exitCode=60 Oct 31 00:45:03.438165 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.437888 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" event={"ID":"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b","Type":"ContainerDied","Data":"0dac6e9b79d5e126bdd77f6698f33ab0cf88636ab4b1d2d6413b517ce5f960e6"} Oct 31 00:45:03.438165 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.437955 5719 scope.go:122] "RemoveContainer" containerID="1329c1743bb6fe172aa491dcd9979ac3552bf9185310ff3ea2856bc246373770" Oct 31 00:45:03.438279 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.438197 5719 scope.go:122] "RemoveContainer" containerID="0dac6e9b79d5e126bdd77f6698f33ab0cf88636ab4b1d2d6413b517ce5f960e6" Oct 31 00:45:03.438520 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:03.438492 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9_odh-ai-gateway-infra(ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" Oct 31 00:45:03.440415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.440392 5719 generic.go:356] "Generic (PLEG): container finished" podID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerID="85b4a31080ea7bab439888b91d918241087c8fb99836a7a4ad2ecdc14aafcb6f" exitCode=60 Oct 31 00:45:03.440532 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.440473 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" event={"ID":"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c","Type":"ContainerDied","Data":"85b4a31080ea7bab439888b91d918241087c8fb99836a7a4ad2ecdc14aafcb6f"} Oct 31 00:45:03.441048 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.441027 5719 scope.go:122] "RemoveContainer" containerID="85b4a31080ea7bab439888b91d918241087c8fb99836a7a4ad2ecdc14aafcb6f" Oct 31 00:45:03.441316 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:03.441293 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-5pqwk_odh-ai-gateway-infra(0ea5a75a-316c-4cbe-ada8-2a2c9495e10c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" Oct 31 00:45:03.443011 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.442981 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" event={"ID":"5fc8901e-25ef-4430-b35c-17556e95d517","Type":"ContainerStarted","Data":"b027dd3b5eb662161a4ca270816402008062e5b0f8caea2fb63b3551a9ba9fb1"} Oct 31 00:45:03.445612 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.445590 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" event={"ID":"237897e0-a086-41fd-b910-a82883ace527","Type":"ContainerStarted","Data":"19097ce7a253434202b868b0cb2fbf3e23175c0115ff310e4b6f467781218214"} Oct 31 00:45:03.464967 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.464944 5719 scope.go:122] "RemoveContainer" containerID="7ecb2f0f69d6bf0eda0c7ef5af07aa1f8750ed11367f60bc0e794ac2da86d291" Oct 31 00:45:03.664078 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:03.664021 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" podStartSLOduration=3.664001376 podStartE2EDuration="3.664001376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:03.661294 +0000 UTC m=+985.197436306" watchObservedRunningTime="2026-10-31 00:45:03.664001376 +0000 UTC m=+985.200143683" Oct 31 00:45:04.173848 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.173812 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:45:04.174051 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.173868 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:45:04.248619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.248587 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:45:04.249181 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.249160 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:45:04.250158 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.250135 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 00:45:04.453415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.453345 5719 scope.go:122] "RemoveContainer" containerID="85b4a31080ea7bab439888b91d918241087c8fb99836a7a4ad2ecdc14aafcb6f" Oct 31 00:45:04.453843 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:04.453600 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-5pqwk_odh-ai-gateway-infra(0ea5a75a-316c-4cbe-ada8-2a2c9495e10c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" Oct 31 00:45:04.454844 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.454822 5719 generic.go:356] "Generic (PLEG): container finished" podID="5fc8901e-25ef-4430-b35c-17556e95d517" containerID="b027dd3b5eb662161a4ca270816402008062e5b0f8caea2fb63b3551a9ba9fb1" exitCode=60 Oct 31 00:45:04.454963 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.454896 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" event={"ID":"5fc8901e-25ef-4430-b35c-17556e95d517","Type":"ContainerDied","Data":"b027dd3b5eb662161a4ca270816402008062e5b0f8caea2fb63b3551a9ba9fb1"} Oct 31 00:45:04.454963 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.454952 5719 scope.go:122] "RemoveContainer" containerID="a88696fab083a2cc4516667262155de5ec9d89bdbb70082e9c4e062f7eed162e" Oct 31 00:45:04.455116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.455100 5719 scope.go:122] "RemoveContainer" containerID="b027dd3b5eb662161a4ca270816402008062e5b0f8caea2fb63b3551a9ba9fb1" Oct 31 00:45:04.455368 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:04.455347 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb_odh-ai-gateway-infra(5fc8901e-25ef-4430-b35c-17556e95d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" Oct 31 00:45:04.456865 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.456840 5719 generic.go:356] "Generic (PLEG): container finished" podID="237897e0-a086-41fd-b910-a82883ace527" containerID="19097ce7a253434202b868b0cb2fbf3e23175c0115ff310e4b6f467781218214" exitCode=60 Oct 31 00:45:04.456974 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.456923 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" event={"ID":"237897e0-a086-41fd-b910-a82883ace527","Type":"ContainerDied","Data":"19097ce7a253434202b868b0cb2fbf3e23175c0115ff310e4b6f467781218214"} Oct 31 00:45:04.457133 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.457118 5719 scope.go:122] "RemoveContainer" containerID="19097ce7a253434202b868b0cb2fbf3e23175c0115ff310e4b6f467781218214" Oct 31 00:45:04.457308 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:04.457290 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd_odh-ai-gateway-infra(237897e0-a086-41fd-b910-a82883ace527)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" podUID="237897e0-a086-41fd-b910-a82883ace527" Oct 31 00:45:04.458747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.458730 5719 scope.go:122] "RemoveContainer" containerID="0dac6e9b79d5e126bdd77f6698f33ab0cf88636ab4b1d2d6413b517ce5f960e6" Oct 31 00:45:04.458951 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:04.458933 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9_odh-ai-gateway-infra(ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" Oct 31 00:45:04.467087 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:04.467072 5719 scope.go:122] "RemoveContainer" containerID="a938e11a496319c0e6dc8bb81af8921665778c6059eadec77ddbc7d313097cfb" Oct 31 00:45:05.466158 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:05.466131 5719 scope.go:122] "RemoveContainer" containerID="b027dd3b5eb662161a4ca270816402008062e5b0f8caea2fb63b3551a9ba9fb1" Oct 31 00:45:05.466616 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:05.466399 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb_odh-ai-gateway-infra(5fc8901e-25ef-4430-b35c-17556e95d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" Oct 31 00:45:05.467903 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:05.467880 5719 scope.go:122] "RemoveContainer" containerID="19097ce7a253434202b868b0cb2fbf3e23175c0115ff310e4b6f467781218214" Oct 31 00:45:05.468113 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:05.468094 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd_odh-ai-gateway-infra(237897e0-a086-41fd-b910-a82883ace527)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" podUID="237897e0-a086-41fd-b910-a82883ace527" Oct 31 00:45:08.574355 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:08.574322 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:45:14.594937 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:14.594906 5719 scope.go:122] "RemoveContainer" containerID="85b4a31080ea7bab439888b91d918241087c8fb99836a7a4ad2ecdc14aafcb6f" Oct 31 00:45:15.536609 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:15.536579 5719 generic.go:356] "Generic (PLEG): container finished" podID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerID="662d8ac1a32df6b721d6cac834ce74ba746db4326e7dde724d0917318c3322f4" exitCode=60 Oct 31 00:45:15.536753 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:15.536633 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" event={"ID":"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c","Type":"ContainerDied","Data":"662d8ac1a32df6b721d6cac834ce74ba746db4326e7dde724d0917318c3322f4"} Oct 31 00:45:15.536753 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:15.536677 5719 scope.go:122] "RemoveContainer" containerID="85b4a31080ea7bab439888b91d918241087c8fb99836a7a4ad2ecdc14aafcb6f" Oct 31 00:45:15.537034 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:15.537010 5719 scope.go:122] "RemoveContainer" containerID="662d8ac1a32df6b721d6cac834ce74ba746db4326e7dde724d0917318c3322f4" Oct 31 00:45:15.537271 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:15.537252 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890125-5pqwk_odh-ai-gateway-infra(0ea5a75a-316c-4cbe-ada8-2a2c9495e10c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" Oct 31 00:45:15.595404 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:15.595385 5719 scope.go:122] "RemoveContainer" containerID="19097ce7a253434202b868b0cb2fbf3e23175c0115ff310e4b6f467781218214" Oct 31 00:45:16.545171 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.545147 5719 generic.go:356] "Generic (PLEG): container finished" podID="237897e0-a086-41fd-b910-a82883ace527" containerID="9a96cc4e5583450adfd500a14e74ac41a39978c9074216f2e7fa32f52828511f" exitCode=60 Oct 31 00:45:16.545283 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.545225 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" event={"ID":"237897e0-a086-41fd-b910-a82883ace527","Type":"ContainerDied","Data":"9a96cc4e5583450adfd500a14e74ac41a39978c9074216f2e7fa32f52828511f"} Oct 31 00:45:16.545337 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.545287 5719 scope.go:122] "RemoveContainer" containerID="19097ce7a253434202b868b0cb2fbf3e23175c0115ff310e4b6f467781218214" Oct 31 00:45:16.545609 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.545581 5719 scope.go:122] "RemoveContainer" containerID="9a96cc4e5583450adfd500a14e74ac41a39978c9074216f2e7fa32f52828511f" Oct 31 00:45:16.545851 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:16.545832 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd_odh-ai-gateway-infra(237897e0-a086-41fd-b910-a82883ace527)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" podUID="237897e0-a086-41fd-b910-a82883ace527" Oct 31 00:45:16.594848 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.594827 5719 scope.go:122] "RemoveContainer" containerID="0dac6e9b79d5e126bdd77f6698f33ab0cf88636ab4b1d2d6413b517ce5f960e6" Oct 31 00:45:16.658465 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.658441 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk"] Oct 31 00:45:16.802309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.802286 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:16.898676 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.898654 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-maas-api-service-ca\") pod \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\" (UID: \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\") " Oct 31 00:45:16.898823 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.898757 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-kube-api-access-p62nn\" (UniqueName: \"kubernetes.io/projected/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-kube-api-access-p62nn\") pod \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\" (UID: \"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c\") " Oct 31 00:45:16.898968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.898947 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-maas-api-service-ca" pod "0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" (UID: "0ea5a75a-316c-4cbe-ada8-2a2c9495e10c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:16.899211 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.899188 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:16.900977 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:16.900929 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-kube-api-access-p62nn" pod "0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" (UID: "0ea5a75a-316c-4cbe-ada8-2a2c9495e10c"). InnerVolumeSpecName "kube-api-access-p62nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:17.000201 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.000134 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p62nn\" (UniqueName: \"kubernetes.io/projected/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c-kube-api-access-p62nn\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:17.552611 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.552585 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" Oct 31 00:45:17.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.552586 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk" event={"ID":"0ea5a75a-316c-4cbe-ada8-2a2c9495e10c","Type":"ContainerDied","Data":"461b7e936b3e7b324075eab0f23790656ecf39a8ad2af72b051e5856fe57a6a8"} Oct 31 00:45:17.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.552742 5719 scope.go:122] "RemoveContainer" containerID="662d8ac1a32df6b721d6cac834ce74ba746db4326e7dde724d0917318c3322f4" Oct 31 00:45:17.556785 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.556748 5719 generic.go:356] "Generic (PLEG): container finished" podID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerID="992295ab756120cb3b8958339c855e5b8ce1b43567d0e88608261793238edb5b" exitCode=60 Oct 31 00:45:17.556785 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.556786 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" event={"ID":"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b","Type":"ContainerDied","Data":"992295ab756120cb3b8958339c855e5b8ce1b43567d0e88608261793238edb5b"} Oct 31 00:45:17.557166 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.557150 5719 scope.go:122] "RemoveContainer" containerID="992295ab756120cb3b8958339c855e5b8ce1b43567d0e88608261793238edb5b" Oct 31 00:45:17.557411 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:17.557390 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9_odh-ai-gateway-infra(ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" Oct 31 00:45:17.562768 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.562750 5719 scope.go:122] "RemoveContainer" containerID="0dac6e9b79d5e126bdd77f6698f33ab0cf88636ab4b1d2d6413b517ce5f960e6" Oct 31 00:45:17.595148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.595132 5719 scope.go:122] "RemoveContainer" containerID="b027dd3b5eb662161a4ca270816402008062e5b0f8caea2fb63b3551a9ba9fb1" Oct 31 00:45:17.669851 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.669794 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd"] Oct 31 00:45:17.711725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.711672 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk"] Oct 31 00:45:17.716492 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.716467 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890125-5pqwk"] Oct 31 00:45:17.827418 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.827393 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:17.910324 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.910299 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/237897e0-a086-41fd-b910-a82883ace527-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237897e0-a086-41fd-b910-a82883ace527-maas-api-service-ca\") pod \"237897e0-a086-41fd-b910-a82883ace527\" (UID: \"237897e0-a086-41fd-b910-a82883ace527\") " Oct 31 00:45:17.910450 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.910371 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/237897e0-a086-41fd-b910-a82883ace527-kube-api-access-5br5j\" (UniqueName: \"kubernetes.io/projected/237897e0-a086-41fd-b910-a82883ace527-kube-api-access-5br5j\") pod \"237897e0-a086-41fd-b910-a82883ace527\" (UID: \"237897e0-a086-41fd-b910-a82883ace527\") " Oct 31 00:45:17.910614 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.910594 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/237897e0-a086-41fd-b910-a82883ace527-maas-api-service-ca" pod "237897e0-a086-41fd-b910-a82883ace527" (UID: "237897e0-a086-41fd-b910-a82883ace527"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:17.912479 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:17.912460 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/237897e0-a086-41fd-b910-a82883ace527-kube-api-access-5br5j" pod "237897e0-a086-41fd-b910-a82883ace527" (UID: "237897e0-a086-41fd-b910-a82883ace527"). InnerVolumeSpecName "kube-api-access-5br5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:18.011916 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.011848 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/237897e0-a086-41fd-b910-a82883ace527-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.011916 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.011874 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5br5j\" (UniqueName: \"kubernetes.io/projected/237897e0-a086-41fd-b910-a82883ace527-kube-api-access-5br5j\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:18.566358 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.566321 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" event={"ID":"5fc8901e-25ef-4430-b35c-17556e95d517","Type":"ContainerStarted","Data":"3005d90f96a4b642e932c39f35a5cbc9976fc4529c03a6970f3c4b0a8f879293"} Oct 31 00:45:18.567888 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.567860 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" event={"ID":"237897e0-a086-41fd-b910-a82883ace527","Type":"ContainerDied","Data":"3845f818b60040500ee41d9314f332ce6db4a41511e6578c9a9d29a4653ec1be"} Oct 31 00:45:18.567888 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.567880 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd" Oct 31 00:45:18.568086 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.567924 5719 scope.go:122] "RemoveContainer" containerID="9a96cc4e5583450adfd500a14e74ac41a39978c9074216f2e7fa32f52828511f" Oct 31 00:45:18.600970 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.600948 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" path="/var/lib/kubelet/pods/0ea5a75a-316c-4cbe-ada8-2a2c9495e10c/volumes" Oct 31 00:45:18.786287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.786262 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd"] Oct 31 00:45:18.790534 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.790511 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890125-2cfmd"] Oct 31 00:45:18.836275 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.836222 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9"] Oct 31 00:45:18.975373 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:18.975349 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:19.023812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.023789 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-kube-api-access-8d9kj\" (UniqueName: \"kubernetes.io/projected/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-kube-api-access-8d9kj\") pod \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\" (UID: \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\") " Oct 31 00:45:19.024039 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.024018 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-maas-api-service-ca\") pod \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\" (UID: \"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b\") " Oct 31 00:45:19.024385 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.024360 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-maas-api-service-ca" pod "ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" (UID: "ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:19.026396 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.026373 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-kube-api-access-8d9kj" pod "ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" (UID: "ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b"). InnerVolumeSpecName "kube-api-access-8d9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:19.125110 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.125041 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.125110 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.125066 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d9kj\" (UniqueName: \"kubernetes.io/projected/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b-kube-api-access-8d9kj\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:19.577062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.577036 5719 generic.go:356] "Generic (PLEG): container finished" podID="5fc8901e-25ef-4430-b35c-17556e95d517" containerID="3005d90f96a4b642e932c39f35a5cbc9976fc4529c03a6970f3c4b0a8f879293" exitCode=60 Oct 31 00:45:19.577241 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.577117 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" event={"ID":"5fc8901e-25ef-4430-b35c-17556e95d517","Type":"ContainerDied","Data":"3005d90f96a4b642e932c39f35a5cbc9976fc4529c03a6970f3c4b0a8f879293"} Oct 31 00:45:19.577241 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.577175 5719 scope.go:122] "RemoveContainer" containerID="b027dd3b5eb662161a4ca270816402008062e5b0f8caea2fb63b3551a9ba9fb1" Oct 31 00:45:19.577470 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.577445 5719 scope.go:122] "RemoveContainer" containerID="3005d90f96a4b642e932c39f35a5cbc9976fc4529c03a6970f3c4b0a8f879293" Oct 31 00:45:19.577724 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:19.577676 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb_odh-ai-gateway-infra(5fc8901e-25ef-4430-b35c-17556e95d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" Oct 31 00:45:19.579406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.579386 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" Oct 31 00:45:19.579490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.579413 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9" event={"ID":"ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b","Type":"ContainerDied","Data":"f7edcad35c9dee7f2930a73a2e15eed7e927ed16cf7ca29bc4b070b22a71dd81"} Oct 31 00:45:19.589120 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.589103 5719 scope.go:122] "RemoveContainer" containerID="992295ab756120cb3b8958339c855e5b8ce1b43567d0e88608261793238edb5b" Oct 31 00:45:19.688319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.688297 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb"] Oct 31 00:45:19.806305 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.806282 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9"] Oct 31 00:45:19.810384 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:19.810362 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890125-qnvc9"] Oct 31 00:45:20.350420 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350392 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj"] Oct 31 00:45:20.350829 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350816 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:20.350886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350830 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:20.350886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350842 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:20.350886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350848 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:20.350886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350857 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:20.350886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350862 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:20.350886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350876 5719 cpu_manager.go:414] "containerMap: removing container" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:20.350886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350880 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350907 5719 cpu_manager.go:414] "containerMap: removing container" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350912 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350924 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350928 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350989 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.350996 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.351002 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.351008 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.351012 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.351016 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.351021 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.351026 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:20.351116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.351033 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:20.354066 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.354049 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.404736 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.404710 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd98f0-openshift-default-dockercfg-6r29c\"" Oct 31 00:45:20.419722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.413628 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj"] Oct 31 00:45:20.437650 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.437628 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdww\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-kube-api-access-qsdww\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.437787 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.437701 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.437849 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.437782 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.437901 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.437847 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.438007 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.437981 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.438056 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.438025 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.438110 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.438088 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.438163 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.438125 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.438237 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.438219 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539220 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539252 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539418 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539270 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539418 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539381 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539548 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539452 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539548 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539516 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdww\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-kube-api-access-qsdww\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539648 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539595 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539648 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539612 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539792 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539644 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539792 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539740 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539928 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539815 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.539928 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539840 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.540026 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.539965 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.540259 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.540238 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.541609 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.541589 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.542105 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.542088 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.598538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.598514 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.598882 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.598859 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdww\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-kube-api-access-qsdww\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.601164 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.601109 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="237897e0-a086-41fd-b910-a82883ace527" path="/var/lib/kubelet/pods/237897e0-a086-41fd-b910-a82883ace527/volumes" Oct 31 00:45:20.601528 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.601512 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" path="/var/lib/kubelet/pods/ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b/volumes" Oct 31 00:45:20.667081 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.667053 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:20.719375 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.719351 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:20.843340 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.843308 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc8901e-25ef-4430-b35c-17556e95d517-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8901e-25ef-4430-b35c-17556e95d517-maas-api-service-ca\") pod \"5fc8901e-25ef-4430-b35c-17556e95d517\" (UID: \"5fc8901e-25ef-4430-b35c-17556e95d517\") " Oct 31 00:45:20.843812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.843391 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc8901e-25ef-4430-b35c-17556e95d517-kube-api-access-f2kf8\" (UniqueName: \"kubernetes.io/projected/5fc8901e-25ef-4430-b35c-17556e95d517-kube-api-access-f2kf8\") pod \"5fc8901e-25ef-4430-b35c-17556e95d517\" (UID: \"5fc8901e-25ef-4430-b35c-17556e95d517\") " Oct 31 00:45:20.843812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.843673 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc8901e-25ef-4430-b35c-17556e95d517-maas-api-service-ca" pod "5fc8901e-25ef-4430-b35c-17556e95d517" (UID: "5fc8901e-25ef-4430-b35c-17556e95d517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:20.843812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.843794 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8901e-25ef-4430-b35c-17556e95d517-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:20.845800 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.845772 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc8901e-25ef-4430-b35c-17556e95d517-kube-api-access-f2kf8" pod "5fc8901e-25ef-4430-b35c-17556e95d517" (UID: "5fc8901e-25ef-4430-b35c-17556e95d517"). InnerVolumeSpecName "kube-api-access-f2kf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:20.944971 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:20.944917 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2kf8\" (UniqueName: \"kubernetes.io/projected/5fc8901e-25ef-4430-b35c-17556e95d517-kube-api-access-f2kf8\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:21.095827 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.095791 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj"] Oct 31 00:45:21.101035 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:45:21.101012 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fb9cdd_4cd8_4ecd_b29c_69a6605ff1fd.slice/crio-cfd4bd302a4ddc312f740e15b72157a5b4242e824b5d40a9c94cc8bb06c59006 WatchSource:0}: Error finding container cfd4bd302a4ddc312f740e15b72157a5b4242e824b5d40a9c94cc8bb06c59006: Status 404 returned error can't find the container with id cfd4bd302a4ddc312f740e15b72157a5b4242e824b5d40a9c94cc8bb06c59006 Oct 31 00:45:21.103307 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.103277 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:21.103363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.103342 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:21.103406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.103373 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:21.595030 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.594997 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" Oct 31 00:45:21.595030 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.595028 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb" event={"ID":"5fc8901e-25ef-4430-b35c-17556e95d517","Type":"ContainerDied","Data":"10779fa343c80c9e2474d088306bd81d4762d6a87842cbf059737e8314cc9b97"} Oct 31 00:45:21.595305 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.595077 5719 scope.go:122] "RemoveContainer" containerID="3005d90f96a4b642e932c39f35a5cbc9976fc4529c03a6970f3c4b0a8f879293" Oct 31 00:45:21.597208 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.597180 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" event={"ID":"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd","Type":"ContainerStarted","Data":"aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9"} Oct 31 00:45:21.597315 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.597230 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" event={"ID":"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd","Type":"ContainerStarted","Data":"cfd4bd302a4ddc312f740e15b72157a5b4242e824b5d40a9c94cc8bb06c59006"} Oct 31 00:45:21.668085 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.668058 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:21.763338 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.763278 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" podStartSLOduration=1.763259707 podStartE2EDuration="1.763259707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:21.762272786 +0000 UTC m=+1003.298415106" watchObservedRunningTime="2026-10-31 00:45:21.763259707 +0000 UTC m=+1003.299402013" Oct 31 00:45:21.878058 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.878027 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb"] Oct 31 00:45:21.881538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:21.881511 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890125-s7knb"] Oct 31 00:45:22.621719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:22.621026 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" path="/var/lib/kubelet/pods/5fc8901e-25ef-4430-b35c-17556e95d517/volumes" Oct 31 00:45:22.668648 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:22.668622 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:45:22.668811 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:22.668675 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:45:23.667922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:23.667892 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:45:23.668299 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:23.667941 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:45:24.668085 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:24.668039 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:45:24.668600 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:24.668163 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:45:25.667922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:25.667891 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:45:25.668121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:25.667950 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:45:26.668317 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:26.668290 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:45:26.668779 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:26.668754 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 00:45:27.704591 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:27.704557 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:27.706078 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:27.706046 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:27.707309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:27.707288 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 00:45:30.050820 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.050785 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f"] Oct 31 00:45:30.051256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051243 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:45:30.051302 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051257 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:45:30.051302 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051274 5719 cpu_manager.go:414] "containerMap: removing container" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:30.051302 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051279 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="237897e0-a086-41fd-b910-a82883ace527" containerName="cleanup" Oct 31 00:45:30.051302 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051297 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:30.051302 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051302 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea5a75a-316c-4cbe-ada8-2a2c9495e10c" containerName="cleanup" Oct 31 00:45:30.051490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051312 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:45:30.051490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051317 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:45:30.051490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051324 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:30.051490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051328 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3d9e07-f23d-4d5c-95a8-d5f434b0d41b" containerName="cleanup" Oct 31 00:45:30.051490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051431 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:45:30.051490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.051446 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:45:30.056041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.056024 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.106576 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.106552 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-29ece5-openshift-default-dockercfg-6vh8l\"" Oct 31 00:45:30.157449 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.157419 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f"] Oct 31 00:45:30.164768 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.164739 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.164892 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.164776 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.164892 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.164807 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.164965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.164903 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.164965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.164931 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvb9\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-kube-api-access-thvb9\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.165035 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.165023 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.165086 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.165073 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5608df70-fa36-4805-8893-1277b632649a-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.165120 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.165094 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5608df70-fa36-4805-8893-1277b632649a-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.165221 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.165205 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.266764 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.266727 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5608df70-fa36-4805-8893-1277b632649a-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.266764 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.266767 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5608df70-fa36-4805-8893-1277b632649a-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.266987 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.266852 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.266987 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.266903 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.266987 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.266933 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.266987 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.266979 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.267200 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.267020 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.267200 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.267046 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvb9\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-kube-api-access-thvb9\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.267200 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.267122 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.267346 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.267274 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.267404 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.267352 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.267509 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.267484 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.267610 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.267578 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5608df70-fa36-4805-8893-1277b632649a-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.267811 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.267713 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.269391 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.269372 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.269455 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.269417 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5608df70-fa36-4805-8893-1277b632649a-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.325980 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.325957 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.326140 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.326123 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvb9\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-kube-api-access-thvb9\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.368812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.368787 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:30.794660 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.794585 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f"] Oct 31 00:45:30.799885 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:45:30.799861 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5608df70_fa36_4805_8893_1277b632649a.slice/crio-4c3380c4b06687cfec8619e3039b0c62a05f275d66957026c78feef73d394865 WatchSource:0}: Error finding container 4c3380c4b06687cfec8619e3039b0c62a05f275d66957026c78feef73d394865: Status 404 returned error can't find the container with id 4c3380c4b06687cfec8619e3039b0c62a05f275d66957026c78feef73d394865 Oct 31 00:45:30.802139 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.802106 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:30.802231 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.802173 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:30.802231 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:30.802201 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:31.679429 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:31.679392 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" event={"ID":"5608df70-fa36-4805-8893-1277b632649a","Type":"ContainerStarted","Data":"1f37b8b9a3d613e055cf0caa5767e95d1fa1850509cb90c3b4836ef1e3f02df7"} Oct 31 00:45:31.679857 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:31.679458 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" event={"ID":"5608df70-fa36-4805-8893-1277b632649a","Type":"ContainerStarted","Data":"4c3380c4b06687cfec8619e3039b0c62a05f275d66957026c78feef73d394865"} Oct 31 00:45:31.844034 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:31.843981 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" podStartSLOduration=2.8439669260000002 podStartE2EDuration="2.843966926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:31.84330218 +0000 UTC m=+1013.379444487" watchObservedRunningTime="2026-10-31 00:45:31.843966926 +0000 UTC m=+1013.380109230" Oct 31 00:45:32.368961 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:32.368931 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:32.370590 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:32.370565 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:45:32.370739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:32.370616 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:45:33.030270 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:45:33.030232 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:45:33.369781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:33.369746 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:45:33.369948 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:33.369805 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:45:34.338789 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:34.338748 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:45:34.369936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:34.369908 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:45:34.370079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:34.369952 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:45:35.369290 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:35.369257 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:45:35.369725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:35.369318 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:45:36.369966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:36.369933 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 00:45:36.370364 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:36.369996 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 00:45:37.697740 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:37.697684 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:37.698208 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:37.698113 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:45:37.698995 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:45:37.698971 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 00:46:01.211936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:01.211902 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:46:03.274643 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:46:03.274600 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:46:24.817374 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.817340 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq"] Oct 31 00:46:24.818061 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.818043 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:46:24.818107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.818064 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:46:24.818187 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.818177 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc8901e-25ef-4430-b35c-17556e95d517" containerName="cleanup" Oct 31 00:46:24.821876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.821859 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:24.902661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.902631 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-home\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:24.902845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.902788 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvxf\" (UniqueName: \"kubernetes.io/projected/f591588a-1257-4c5e-a312-63cdcac0d11d-kube-api-access-nvvxf\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:24.902845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.902815 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-dshm\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:24.902941 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.902907 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f591588a-1257-4c5e-a312-63cdcac0d11d-tls-certs\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:24.902990 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.902950 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-tmp-dir\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:24.902990 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.902966 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-model-cache\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:24.920963 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.920939 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8bec73\"/\"distinct-2-w0-8bec73-kserve-self-signed-certs\"" Oct 31 00:46:24.923367 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:24.923344 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq"] Oct 31 00:46:25.003877 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.003842 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-tmp-dir\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.003877 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.003874 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-model-cache\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.004150 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.003941 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-home\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.004150 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.004030 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvxf\" (UniqueName: \"kubernetes.io/projected/f591588a-1257-4c5e-a312-63cdcac0d11d-kube-api-access-nvvxf\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.004150 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.004081 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-dshm\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.004327 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.004311 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f591588a-1257-4c5e-a312-63cdcac0d11d-tls-certs\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.004385 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.004328 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-tmp-dir\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.004385 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.004354 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-model-cache\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.004486 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.004407 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-home\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.006524 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.006500 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-dshm\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.006819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.006803 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f591588a-1257-4c5e-a312-63cdcac0d11d-tls-certs\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.064197 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.064167 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvxf\" (UniqueName: \"kubernetes.io/projected/f591588a-1257-4c5e-a312-63cdcac0d11d-kube-api-access-nvvxf\") pod \"distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.134913 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.134837 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:25.517788 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:25.517722 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq"] Oct 31 00:46:25.522907 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:46:25.522880 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf591588a_1257_4c5e_a312_63cdcac0d11d.slice/crio-e76b4e568e639d299bb920df82269a94e75ebf12b77d98d7e706be19c3121b6b WatchSource:0}: Error finding container e76b4e568e639d299bb920df82269a94e75ebf12b77d98d7e706be19c3121b6b: Status 404 returned error can't find the container with id e76b4e568e639d299bb920df82269a94e75ebf12b77d98d7e706be19c3121b6b Oct 31 00:46:26.234480 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:26.234447 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" event={"ID":"f591588a-1257-4c5e-a312-63cdcac0d11d","Type":"ContainerStarted","Data":"d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936"} Oct 31 00:46:26.234879 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:26.234493 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" event={"ID":"f591588a-1257-4c5e-a312-63cdcac0d11d","Type":"ContainerStarted","Data":"e76b4e568e639d299bb920df82269a94e75ebf12b77d98d7e706be19c3121b6b"} Oct 31 00:46:26.347780 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:26.347731 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podStartSLOduration=2.347717589 podStartE2EDuration="2.347717589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:26.346918259 +0000 UTC m=+1067.883060565" watchObservedRunningTime="2026-10-31 00:46:26.347717589 +0000 UTC m=+1067.883859895" Oct 31 00:46:27.135482 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:27.135446 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:29.934446 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:29.934412 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9"] Oct 31 00:46:29.938240 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:29.938223 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.041901 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.041858 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9"] Oct 31 00:46:30.063650 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.063617 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.063837 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.063669 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.063883 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.063851 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdfc\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-kube-api-access-4kdfc\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.063926 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.063910 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39984c71-6603-47a3-9344-7407edf3bcd2-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.064008 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.063992 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39984c71-6603-47a3-9344-7407edf3bcd2-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.064046 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.064025 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.064095 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.064054 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.064204 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.064187 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.064261 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.064238 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.164909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.164877 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdfc\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-kube-api-access-4kdfc\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.164930 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39984c71-6603-47a3-9344-7407edf3bcd2-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.164975 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39984c71-6603-47a3-9344-7407edf3bcd2-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165000 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165027 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165293 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165096 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165293 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165208 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165395 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165300 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165395 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165343 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165501 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165650 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165627 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165737 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165664 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165737 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165667 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.165870 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.165791 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39984c71-6603-47a3-9344-7407edf3bcd2-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.167924 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.167904 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39984c71-6603-47a3-9344-7407edf3bcd2-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.168085 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.168063 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.230543 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.230486 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.243307 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.243283 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdfc\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-kube-api-access-4kdfc\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.251462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.251443 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:30.679361 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.679322 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9"] Oct 31 00:46:30.684487 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:46:30.684462 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39984c71_6603_47a3_9344_7407edf3bcd2.slice/crio-a5156ad497d9df9b1bc8ec7f6de88d59de5f67e05a4c2314afeb00aff4bbc9cc WatchSource:0}: Error finding container a5156ad497d9df9b1bc8ec7f6de88d59de5f67e05a4c2314afeb00aff4bbc9cc: Status 404 returned error can't find the container with id a5156ad497d9df9b1bc8ec7f6de88d59de5f67e05a4c2314afeb00aff4bbc9cc Oct 31 00:46:30.690204 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.690172 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:46:30.690287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.690239 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:46:30.690287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:30.690267 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:46:31.262903 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:31.262870 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" event={"ID":"39984c71-6603-47a3-9344-7407edf3bcd2","Type":"ContainerStarted","Data":"226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be"} Oct 31 00:46:31.263296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:31.262927 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" event={"ID":"39984c71-6603-47a3-9344-7407edf3bcd2","Type":"ContainerStarted","Data":"a5156ad497d9df9b1bc8ec7f6de88d59de5f67e05a4c2314afeb00aff4bbc9cc"} Oct 31 00:46:31.380139 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:31.380073 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" podStartSLOduration=2.380054274 podStartE2EDuration="2.380054274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:31.378830421 +0000 UTC m=+1072.914972726" watchObservedRunningTime="2026-10-31 00:46:31.380054274 +0000 UTC m=+1072.916196584" Oct 31 00:46:32.252026 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:32.251992 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:32.253800 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:32.253774 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:32.253939 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:32.253827 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:33.032803 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:46:33.032766 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:46:33.252403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:33.252369 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:33.252572 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:33.252420 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:33.721721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:33.721054 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:46:34.259719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:34.252105 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:34.259719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:34.252194 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:35.136034 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:35.136001 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:35.148450 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:35.148427 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:35.252401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:35.252367 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:35.252558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:35.252424 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:35.296152 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:35.296115 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:46:36.252476 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:36.252444 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:36.252657 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:36.252505 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:37.252282 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:37.252247 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 00:46:37.252720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:37.252297 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 00:46:38.359784 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.359754 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:38.360465 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.360431 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:38.362172 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.362151 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 00:46:38.765230 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.765161 5719 scope.go:122] "RemoveContainer" containerID="6d43b35d32b21506ce4927460d00610b43b330e24cf1f7c641a00e606d9c0c80" Oct 31 00:46:38.774449 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.774430 5719 scope.go:122] "RemoveContainer" containerID="e40fc7f84123b34f22e54734211a10a5872e7bb700b22bb09515688480e659c7" Oct 31 00:46:38.785515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.785493 5719 scope.go:122] "RemoveContainer" containerID="6da962087ce384577f265b912ece629d3010e51f13270761f654e6ac0d0b4ffb" Oct 31 00:46:38.836998 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.836975 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj"] Oct 31 00:46:38.841966 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.841949 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.883000 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.882975 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-dshm\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.883121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.883022 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tls-certs\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.883164 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.883127 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-model-cache\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.883222 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.883208 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tmp-dir\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.883377 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.883361 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-home\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.883414 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.883399 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr87c\" (UniqueName: \"kubernetes.io/projected/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-kube-api-access-hr87c\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.894425 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.894401 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ef2572\"/\"distinct-w4-ef2572-kserve-self-signed-certs\"" Oct 31 00:46:38.897351 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.897327 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj"] Oct 31 00:46:38.984399 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.984369 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-home\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.984578 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.984420 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr87c\" (UniqueName: \"kubernetes.io/projected/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-kube-api-access-hr87c\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.984578 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.984478 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-dshm\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.984578 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.984513 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tls-certs\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.984578 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.984549 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-model-cache\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.984834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.984644 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tmp-dir\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.984834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.984780 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-home\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.984987 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.984962 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-model-cache\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.985096 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.985043 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tmp-dir\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.986858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.986839 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-dshm\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:38.987093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:38.987077 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tls-certs\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:39.044036 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:39.043965 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr87c\" (UniqueName: \"kubernetes.io/projected/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-kube-api-access-hr87c\") pod \"distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:39.153352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:39.153321 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:39.561810 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:39.561778 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj"] Oct 31 00:46:39.565991 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:46:39.565967 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb09e9548_a2b8_4a34_ab3e_6a5e791fc063.slice/crio-fb8151c6ace1020b8336f214df34ad81efd1237a125bfc3b8c1774fa33bc6f80 WatchSource:0}: Error finding container fb8151c6ace1020b8336f214df34ad81efd1237a125bfc3b8c1774fa33bc6f80: Status 404 returned error can't find the container with id fb8151c6ace1020b8336f214df34ad81efd1237a125bfc3b8c1774fa33bc6f80 Oct 31 00:46:40.321932 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:40.321896 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" event={"ID":"b09e9548-a2b8-4a34-ab3e-6a5e791fc063","Type":"ContainerStarted","Data":"aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570"} Oct 31 00:46:40.322130 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:40.321940 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" event={"ID":"b09e9548-a2b8-4a34-ab3e-6a5e791fc063","Type":"ContainerStarted","Data":"fb8151c6ace1020b8336f214df34ad81efd1237a125bfc3b8c1774fa33bc6f80"} Oct 31 00:46:40.435144 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:40.435098 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podStartSLOduration=2.435085767 podStartE2EDuration="2.435085767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:40.433929732 +0000 UTC m=+1081.970072042" watchObservedRunningTime="2026-10-31 00:46:40.435085767 +0000 UTC m=+1081.971228072" Oct 31 00:46:41.153946 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:41.153910 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:49.154107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:49.154037 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:49.166534 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:49.166515 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:46:49.383835 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:46:49.383809 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:47:01.930316 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:01.924791 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:47:03.009364 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:47:03.009331 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:47:03.297914 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.297842 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9"] Oct 31 00:47:03.303579 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.303557 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.353989 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.353957 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-8bec73-openshift-default-dockercfg-xqgfq\"" Oct 31 00:47:03.404292 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.404262 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9"] Oct 31 00:47:03.464521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464485 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd537fc9-02b4-4392-a012-5d6ce626670f-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.464668 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464568 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.464759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464666 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.464759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464708 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.464759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464730 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.464878 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464858 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.464921 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464876 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.464921 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464890 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.464921 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.464911 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8px\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-kube-api-access-dl8px\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565364 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd537fc9-02b4-4392-a012-5d6ce626670f-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565409 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565454 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565474 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565490 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565554 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565572 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565588 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.565624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565613 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8px\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-kube-api-access-dl8px\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.566026 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.565963 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.566095 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.566039 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.566095 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.566083 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.566199 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.566178 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.566256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.566232 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd537fc9-02b4-4392-a012-5d6ce626670f-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.568096 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.568076 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.568451 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.568432 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.624745 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.624686 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.624825 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.624789 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8px\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-kube-api-access-dl8px\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:03.917072 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:03.917048 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:04.359778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:04.359570 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9"] Oct 31 00:47:04.362710 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:47:04.362663 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd537fc9_02b4_4392_a012_5d6ce626670f.slice/crio-5f943e195bc5c9f1ade290485c6c8f22a7d72a4496e4ce82111d4a8cd39c6a66 WatchSource:0}: Error finding container 5f943e195bc5c9f1ade290485c6c8f22a7d72a4496e4ce82111d4a8cd39c6a66: Status 404 returned error can't find the container with id 5f943e195bc5c9f1ade290485c6c8f22a7d72a4496e4ce82111d4a8cd39c6a66 Oct 31 00:47:04.364935 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:04.364907 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:04.365038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:04.364968 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:04.365038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:04.364998 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:04.477061 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:04.477027 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" event={"ID":"bd537fc9-02b4-4392-a012-5d6ce626670f","Type":"ContainerStarted","Data":"87785b4d8476637f60b486e5f95bfdc7e8ea954807c7e1682e080f83c8f85b7f"} Oct 31 00:47:04.477243 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:04.477089 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" event={"ID":"bd537fc9-02b4-4392-a012-5d6ce626670f","Type":"ContainerStarted","Data":"5f943e195bc5c9f1ade290485c6c8f22a7d72a4496e4ce82111d4a8cd39c6a66"} Oct 31 00:47:04.596576 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:04.596520 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podStartSLOduration=1.5965067849999999 podStartE2EDuration="1.596506785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:04.594258085 +0000 UTC m=+1106.130400390" watchObservedRunningTime="2026-10-31 00:47:04.596506785 +0000 UTC m=+1106.132649089" Oct 31 00:47:04.917306 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:04.917191 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:05.924820 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:05.924783 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:47:05.925427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:05.925389 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:47:06.919039 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:06.919004 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:47:06.919355 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:06.919325 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:47:07.922718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:07.922063 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:47:07.922718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:07.922125 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:47:08.924718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:08.920049 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:47:08.924718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:08.920162 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:47:09.922713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:09.922102 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:47:09.922713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:09.922180 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:47:10.918304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:10.918264 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:47:10.918707 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:10.918336 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 31 00:47:11.922134 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:11.922105 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:11.922608 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:11.922483 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:11.923046 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:11.923029 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 00:47:18.088412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.088385 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg"] Oct 31 00:47:18.092635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.092617 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.160643 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.160610 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.160643 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.160652 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.160940 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.160678 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hk5x\" (UniqueName: \"kubernetes.io/projected/acff01b6-dfc5-4675-9768-4c72ad52070e-kube-api-access-7hk5x\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.192867 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.192838 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:18.192867 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.192836 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:18.193108 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.193033 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:47:18.193108 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.193039 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:18.195612 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.195588 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg"] Oct 31 00:47:18.262113 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.262086 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.262249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.262121 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.262249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.262147 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hk5x\" (UniqueName: \"kubernetes.io/projected/acff01b6-dfc5-4675-9768-4c72ad52070e-kube-api-access-7hk5x\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.262683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.262663 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.264678 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.264656 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.321548 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.321525 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hk5x\" (UniqueName: \"kubernetes.io/projected/acff01b6-dfc5-4675-9768-4c72ad52070e-kube-api-access-7hk5x\") pod \"maas-api-revoke-sub-aa983f3a-qxdpg\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.406581 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.406528 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:18.780750 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:18.780635 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg"] Oct 31 00:47:18.785678 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:47:18.785631 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacff01b6_dfc5_4675_9768_4c72ad52070e.slice/crio-61de93a7c13aa626c35ac16707a60dfad46fe331b357429ece9530328fed94c1 WatchSource:0}: Error finding container 61de93a7c13aa626c35ac16707a60dfad46fe331b357429ece9530328fed94c1: Status 404 returned error can't find the container with id 61de93a7c13aa626c35ac16707a60dfad46fe331b357429ece9530328fed94c1 Oct 31 00:47:19.588881 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:19.588847 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" event={"ID":"acff01b6-dfc5-4675-9768-4c72ad52070e","Type":"ContainerStarted","Data":"937f03afad30a438c3f887ac3897ee07e2e95d6ca5a075ec5ada2e355b9cdcf8"} Oct 31 00:47:19.589292 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:19.588934 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" event={"ID":"acff01b6-dfc5-4675-9768-4c72ad52070e","Type":"ContainerStarted","Data":"61de93a7c13aa626c35ac16707a60dfad46fe331b357429ece9530328fed94c1"} Oct 31 00:47:19.702001 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:19.701955 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" podStartSLOduration=1.7019419679999999 podStartE2EDuration="1.701941968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:19.699383493 +0000 UTC m=+1121.235525800" watchObservedRunningTime="2026-10-31 00:47:19.701941968 +0000 UTC m=+1121.238084272" Oct 31 00:47:20.596191 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:20.596155 5719 generic.go:356] "Generic (PLEG): container finished" podID="acff01b6-dfc5-4675-9768-4c72ad52070e" containerID="937f03afad30a438c3f887ac3897ee07e2e95d6ca5a075ec5ada2e355b9cdcf8" exitCode=0 Oct 31 00:47:20.601173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:20.601144 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" event={"ID":"acff01b6-dfc5-4675-9768-4c72ad52070e","Type":"ContainerDied","Data":"937f03afad30a438c3f887ac3897ee07e2e95d6ca5a075ec5ada2e355b9cdcf8"} Oct 31 00:47:21.745811 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.745788 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:21.800147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.800118 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-xks-ca\") pod \"acff01b6-dfc5-4675-9768-4c72ad52070e\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " Oct 31 00:47:21.800262 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.800220 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-openshift-ca\") pod \"acff01b6-dfc5-4675-9768-4c72ad52070e\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " Oct 31 00:47:21.800387 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.800373 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acff01b6-dfc5-4675-9768-4c72ad52070e-kube-api-access-7hk5x\" (UniqueName: \"kubernetes.io/projected/acff01b6-dfc5-4675-9768-4c72ad52070e-kube-api-access-7hk5x\") pod \"acff01b6-dfc5-4675-9768-4c72ad52070e\" (UID: \"acff01b6-dfc5-4675-9768-4c72ad52070e\") " Oct 31 00:47:21.800604 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.800571 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-openshift-ca" pod "acff01b6-dfc5-4675-9768-4c72ad52070e" (UID: "acff01b6-dfc5-4675-9768-4c72ad52070e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:21.800911 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.800784 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:21.802259 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.802238 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-xks-ca" pod "acff01b6-dfc5-4675-9768-4c72ad52070e" (UID: "acff01b6-dfc5-4675-9768-4c72ad52070e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:21.802363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.802345 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acff01b6-dfc5-4675-9768-4c72ad52070e-kube-api-access-7hk5x" pod "acff01b6-dfc5-4675-9768-4c72ad52070e" (UID: "acff01b6-dfc5-4675-9768-4c72ad52070e"). InnerVolumeSpecName "kube-api-access-7hk5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:21.902396 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.902338 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hk5x\" (UniqueName: \"kubernetes.io/projected/acff01b6-dfc5-4675-9768-4c72ad52070e-kube-api-access-7hk5x\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:21.902396 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:21.902362 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acff01b6-dfc5-4675-9768-4c72ad52070e-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:22.608541 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:22.608505 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" event={"ID":"acff01b6-dfc5-4675-9768-4c72ad52070e","Type":"ContainerDied","Data":"61de93a7c13aa626c35ac16707a60dfad46fe331b357429ece9530328fed94c1"} Oct 31 00:47:22.608749 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:22.608557 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61de93a7c13aa626c35ac16707a60dfad46fe331b357429ece9530328fed94c1" Oct 31 00:47:22.608749 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:22.608512 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg" Oct 31 00:47:26.533046 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:26.532890 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:47:30.780612 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.780567 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl"] Oct 31 00:47:30.781175 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.781128 5719 cpu_manager.go:414] "containerMap: removing container" podUID="acff01b6-dfc5-4675-9768-4c72ad52070e" containerName="revoke-keys" Oct 31 00:47:30.781175 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.781174 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acff01b6-dfc5-4675-9768-4c72ad52070e" containerName="revoke-keys" Oct 31 00:47:30.781361 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.781346 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="acff01b6-dfc5-4675-9768-4c72ad52070e" containerName="revoke-keys" Oct 31 00:47:30.784862 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.784843 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:30.877658 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.877074 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:30.877658 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.877256 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlj56\" (UniqueName: \"kubernetes.io/projected/17057d4f-1632-4c84-ab68-9a1f12d85551-kube-api-access-nlj56\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:30.877658 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.877339 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:30.906722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.904061 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:30.906722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.904157 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:30.906722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.904249 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:47:30.919865 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.919836 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:30.944538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.944007 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl"] Oct 31 00:47:30.978402 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.978380 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:30.979008 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.978988 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlj56\" (UniqueName: \"kubernetes.io/projected/17057d4f-1632-4c84-ab68-9a1f12d85551-kube-api-access-nlj56\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:30.979196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.979185 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:30.979910 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.979803 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:30.981537 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:30.981520 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:31.037760 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:31.037679 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlj56\" (UniqueName: \"kubernetes.io/projected/17057d4f-1632-4c84-ab68-9a1f12d85551-kube-api-access-nlj56\") pod \"maas-api-revoke-sub-0066dae7-ff5nl\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:31.100311 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:31.100287 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:31.473935 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:31.473907 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl"] Oct 31 00:47:31.478825 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:47:31.478800 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17057d4f_1632_4c84_ab68_9a1f12d85551.slice/crio-90d81e851a7494fa2f2c5b5faad9f4c0c01f1ef0392c79066af6157326b2a80f WatchSource:0}: Error finding container 90d81e851a7494fa2f2c5b5faad9f4c0c01f1ef0392c79066af6157326b2a80f: Status 404 returned error can't find the container with id 90d81e851a7494fa2f2c5b5faad9f4c0c01f1ef0392c79066af6157326b2a80f Oct 31 00:47:31.761189 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:31.761119 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" event={"ID":"17057d4f-1632-4c84-ab68-9a1f12d85551","Type":"ContainerStarted","Data":"90d81e851a7494fa2f2c5b5faad9f4c0c01f1ef0392c79066af6157326b2a80f"} Oct 31 00:47:32.768841 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:32.768763 5719 generic.go:356] "Generic (PLEG): container finished" podID="17057d4f-1632-4c84-ab68-9a1f12d85551" containerID="43a1da746396129d0a6ba0d70c0711ca94c5552282bded90457656956b567e9a" exitCode=0 Oct 31 00:47:32.769253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:32.768851 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" event={"ID":"17057d4f-1632-4c84-ab68-9a1f12d85551","Type":"ContainerDied","Data":"43a1da746396129d0a6ba0d70c0711ca94c5552282bded90457656956b567e9a"} Oct 31 00:47:33.011860 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:47:33.011828 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:47:33.924940 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:33.924917 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:34.112974 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.112947 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17057d4f-1632-4c84-ab68-9a1f12d85551-kube-api-access-nlj56\" (UniqueName: \"kubernetes.io/projected/17057d4f-1632-4c84-ab68-9a1f12d85551-kube-api-access-nlj56\") pod \"17057d4f-1632-4c84-ab68-9a1f12d85551\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " Oct 31 00:47:34.113147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.113015 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-xks-ca\") pod \"17057d4f-1632-4c84-ab68-9a1f12d85551\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " Oct 31 00:47:34.113147 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.113112 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-openshift-ca\") pod \"17057d4f-1632-4c84-ab68-9a1f12d85551\" (UID: \"17057d4f-1632-4c84-ab68-9a1f12d85551\") " Oct 31 00:47:34.113466 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.113442 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-openshift-ca" pod "17057d4f-1632-4c84-ab68-9a1f12d85551" (UID: "17057d4f-1632-4c84-ab68-9a1f12d85551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:34.115210 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.115190 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-xks-ca" pod "17057d4f-1632-4c84-ab68-9a1f12d85551" (UID: "17057d4f-1632-4c84-ab68-9a1f12d85551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:34.115275 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.115240 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17057d4f-1632-4c84-ab68-9a1f12d85551-kube-api-access-nlj56" pod "17057d4f-1632-4c84-ab68-9a1f12d85551" (UID: "17057d4f-1632-4c84-ab68-9a1f12d85551"). InnerVolumeSpecName "kube-api-access-nlj56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:34.214297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.214267 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.214297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.214293 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlj56\" (UniqueName: \"kubernetes.io/projected/17057d4f-1632-4c84-ab68-9a1f12d85551-kube-api-access-nlj56\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.214456 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.214304 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17057d4f-1632-4c84-ab68-9a1f12d85551-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:34.788548 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.788523 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" Oct 31 00:47:34.788707 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.788560 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl" event={"ID":"17057d4f-1632-4c84-ab68-9a1f12d85551","Type":"ContainerDied","Data":"90d81e851a7494fa2f2c5b5faad9f4c0c01f1ef0392c79066af6157326b2a80f"} Oct 31 00:47:34.788707 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:34.788607 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d81e851a7494fa2f2c5b5faad9f4c0c01f1ef0392c79066af6157326b2a80f" Oct 31 00:47:47.171271 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.166727 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28"] Oct 31 00:47:47.171271 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.167501 5719 cpu_manager.go:414] "containerMap: removing container" podUID="17057d4f-1632-4c84-ab68-9a1f12d85551" containerName="revoke-keys" Oct 31 00:47:47.171271 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.167525 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17057d4f-1632-4c84-ab68-9a1f12d85551" containerName="revoke-keys" Oct 31 00:47:47.171271 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.167654 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="17057d4f-1632-4c84-ab68-9a1f12d85551" containerName="revoke-keys" Oct 31 00:47:47.172708 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.172664 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.224717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.220103 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28"] Oct 31 00:47:47.261063 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.261029 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:47:47.261412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.261393 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:47:47.261613 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.261596 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:47:47.262413 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.262383 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.262764 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.262740 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbx2b\" (UniqueName: \"kubernetes.io/projected/bf1005dc-1399-4230-b8d7-088a0912056c-kube-api-access-rbx2b\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.262924 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.262907 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.272858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.272837 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:47:47.369718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.369545 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbx2b\" (UniqueName: \"kubernetes.io/projected/bf1005dc-1399-4230-b8d7-088a0912056c-kube-api-access-rbx2b\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.369718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.369613 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.369718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.369720 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.372897 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.371595 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.376639 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.376612 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.469871 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.469802 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbx2b\" (UniqueName: \"kubernetes.io/projected/bf1005dc-1399-4230-b8d7-088a0912056c-kube-api-access-rbx2b\") pod \"maas-api-revoke-sub-4c5bb6bf-d5c28\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:47.513786 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:47.507856 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:48.010289 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:48.009615 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28"] Oct 31 00:47:48.035560 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:47:48.035529 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1005dc_1399_4230_b8d7_088a0912056c.slice/crio-1e5cf0e8db7e36a66c688a6f8d3c28e068993b52e5df8c7331efe40cdde580fe WatchSource:0}: Error finding container 1e5cf0e8db7e36a66c688a6f8d3c28e068993b52e5df8c7331efe40cdde580fe: Status 404 returned error can't find the container with id 1e5cf0e8db7e36a66c688a6f8d3c28e068993b52e5df8c7331efe40cdde580fe Oct 31 00:47:48.893105 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:48.893067 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" event={"ID":"bf1005dc-1399-4230-b8d7-088a0912056c","Type":"ContainerStarted","Data":"945c11e99d491ddeb07e07131d9cc67b750856286bbfd49d461bb105bb99bf3a"} Oct 31 00:47:48.893733 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:48.893680 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" event={"ID":"bf1005dc-1399-4230-b8d7-088a0912056c","Type":"ContainerStarted","Data":"1e5cf0e8db7e36a66c688a6f8d3c28e068993b52e5df8c7331efe40cdde580fe"} Oct 31 00:47:49.017760 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:49.016653 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" podStartSLOduration=2.016634864 podStartE2EDuration="2.016634864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:49.012272332 +0000 UTC m=+1150.548414638" watchObservedRunningTime="2026-10-31 00:47:49.016634864 +0000 UTC m=+1150.552777171" Oct 31 00:47:49.922761 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:49.922728 5719 generic.go:356] "Generic (PLEG): container finished" podID="bf1005dc-1399-4230-b8d7-088a0912056c" containerID="945c11e99d491ddeb07e07131d9cc67b750856286bbfd49d461bb105bb99bf3a" exitCode=0 Oct 31 00:47:49.923348 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:49.923322 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" event={"ID":"bf1005dc-1399-4230-b8d7-088a0912056c","Type":"ContainerDied","Data":"945c11e99d491ddeb07e07131d9cc67b750856286bbfd49d461bb105bb99bf3a"} Oct 31 00:47:50.296868 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:50.296775 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:47:50.296868 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:50.296823 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:47:51.079152 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.079129 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:51.175151 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.175112 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1005dc-1399-4230-b8d7-088a0912056c-kube-api-access-rbx2b\" (UniqueName: \"kubernetes.io/projected/bf1005dc-1399-4230-b8d7-088a0912056c-kube-api-access-rbx2b\") pod \"bf1005dc-1399-4230-b8d7-088a0912056c\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " Oct 31 00:47:51.175295 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.175183 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-openshift-ca\") pod \"bf1005dc-1399-4230-b8d7-088a0912056c\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " Oct 31 00:47:51.175366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.175297 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-xks-ca\") pod \"bf1005dc-1399-4230-b8d7-088a0912056c\" (UID: \"bf1005dc-1399-4230-b8d7-088a0912056c\") " Oct 31 00:47:51.176510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.176479 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-openshift-ca" pod "bf1005dc-1399-4230-b8d7-088a0912056c" (UID: "bf1005dc-1399-4230-b8d7-088a0912056c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:51.177494 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.177465 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-xks-ca" pod "bf1005dc-1399-4230-b8d7-088a0912056c" (UID: "bf1005dc-1399-4230-b8d7-088a0912056c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:51.177568 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.177491 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1005dc-1399-4230-b8d7-088a0912056c-kube-api-access-rbx2b" pod "bf1005dc-1399-4230-b8d7-088a0912056c" (UID: "bf1005dc-1399-4230-b8d7-088a0912056c"). InnerVolumeSpecName "kube-api-access-rbx2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:51.277217 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.277188 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbx2b\" (UniqueName: \"kubernetes.io/projected/bf1005dc-1399-4230-b8d7-088a0912056c-kube-api-access-rbx2b\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:51.277217 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.277216 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:51.277398 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.277230 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1005dc-1399-4230-b8d7-088a0912056c-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:51.935481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.935446 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" Oct 31 00:47:51.935651 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.935479 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28" event={"ID":"bf1005dc-1399-4230-b8d7-088a0912056c","Type":"ContainerDied","Data":"1e5cf0e8db7e36a66c688a6f8d3c28e068993b52e5df8c7331efe40cdde580fe"} Oct 31 00:47:51.935651 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:51.935553 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5cf0e8db7e36a66c688a6f8d3c28e068993b52e5df8c7331efe40cdde580fe" Oct 31 00:47:58.205881 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:58.205846 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:47:59.960452 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:59.958144 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q"] Oct 31 00:47:59.960452 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:59.958791 5719 cpu_manager.go:414] "containerMap: removing container" podUID="bf1005dc-1399-4230-b8d7-088a0912056c" containerName="revoke-keys" Oct 31 00:47:59.960452 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:59.958809 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1005dc-1399-4230-b8d7-088a0912056c" containerName="revoke-keys" Oct 31 00:47:59.960452 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:59.958987 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1005dc-1399-4230-b8d7-088a0912056c" containerName="revoke-keys" Oct 31 00:47:59.962514 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:47:59.962489 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.053157 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.053124 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q"] Oct 31 00:48:00.064481 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.064450 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.064671 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.064513 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.064671 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.064647 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.064811 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.064730 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.064811 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.064769 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.064811 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.064799 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8784813-29fc-46ba-aa14-cc965d025f0f-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.064972 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.064895 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.065027 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.064996 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.065081 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.065040 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-kube-api-access-f7kx2\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166035 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166075 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166101 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166141 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166168 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8784813-29fc-46ba-aa14-cc965d025f0f-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166204 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166533 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166318 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166533 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166346 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-kube-api-access-f7kx2\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166533 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166436 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166533 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166499 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166849 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-credential-socket\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.166979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166874 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8784813-29fc-46ba-aa14-cc965d025f0f-istiod-ca-cert\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.167046 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.166993 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-certs\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.167046 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.167031 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-data\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.168933 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.168911 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-envoy\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.169463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.169434 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-podinfo\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.225732 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.225629 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-token\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.225882 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.225780 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-kube-api-access-f7kx2\") pod \"e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.276507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.276480 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:00.704895 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.704866 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q"] Oct 31 00:48:00.709211 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:00.709187 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8784813_29fc_46ba_aa14_cc965d025f0f.slice/crio-88bba319697acd7b86069eccd346b5399ec444e8a5b2221b7d29f8ca1a66e550 WatchSource:0}: Error finding container 88bba319697acd7b86069eccd346b5399ec444e8a5b2221b7d29f8ca1a66e550: Status 404 returned error can't find the container with id 88bba319697acd7b86069eccd346b5399ec444e8a5b2221b7d29f8ca1a66e550 Oct 31 00:48:00.711256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.711227 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:00.711338 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.711286 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:00.711338 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.711315 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:00.993038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.992943 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" event={"ID":"c8784813-29fc-46ba-aa14-cc965d025f0f","Type":"ContainerStarted","Data":"3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e"} Oct 31 00:48:00.993038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:00.993006 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" event={"ID":"c8784813-29fc-46ba-aa14-cc965d025f0f","Type":"ContainerStarted","Data":"88bba319697acd7b86069eccd346b5399ec444e8a5b2221b7d29f8ca1a66e550"} Oct 31 00:48:01.110801 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:01.110752 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" podStartSLOduration=2.110737343 podStartE2EDuration="2.110737343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:01.109377326 +0000 UTC m=+1162.645519654" watchObservedRunningTime="2026-10-31 00:48:01.110737343 +0000 UTC m=+1162.646879647" Oct 31 00:48:01.277440 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:01.277363 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:01.278756 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:01.278729 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:48:01.278906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:01.278782 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:48:02.277817 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.277773 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:48:02.278214 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.277858 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:48:02.770154 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.770123 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf"] Oct 31 00:48:02.774083 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.774062 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.797012 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.796989 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.797108 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.797024 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.797108 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.797059 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4bk\" (UniqueName: \"kubernetes.io/projected/e3616268-b555-45bb-9d19-ea73c288999d-kube-api-access-rt4bk\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.873887 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.873856 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:02.874043 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.873985 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:02.874043 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.874001 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:02.874121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.874062 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:48:02.876975 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.876952 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf"] Oct 31 00:48:02.897983 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.897956 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.898122 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.898012 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.898122 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.898082 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4bk\" (UniqueName: \"kubernetes.io/projected/e3616268-b555-45bb-9d19-ea73c288999d-kube-api-access-rt4bk\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.898792 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.898770 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.932205 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.932171 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:02.956964 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:02.956932 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4bk\" (UniqueName: \"kubernetes.io/projected/e3616268-b555-45bb-9d19-ea73c288999d-kube-api-access-rt4bk\") pod \"maas-api-revoke-sub-c4185b87-6mwvf\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:03.032594 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:48:03.032433 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:48:03.087923 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:03.087895 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:03.277323 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:03.277288 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:48:03.277459 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:03.277348 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:48:03.524734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:03.520304 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf"] Oct 31 00:48:03.528620 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:03.528589 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3616268_b555_45bb_9d19_ea73c288999d.slice/crio-09d6bd568f9cc7f5889cacf19c414bd11b21c7070f301e1dacd2b1e7f0245ad6 WatchSource:0}: Error finding container 09d6bd568f9cc7f5889cacf19c414bd11b21c7070f301e1dacd2b1e7f0245ad6: Status 404 returned error can't find the container with id 09d6bd568f9cc7f5889cacf19c414bd11b21c7070f301e1dacd2b1e7f0245ad6 Oct 31 00:48:04.033737 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:04.033681 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" event={"ID":"e3616268-b555-45bb-9d19-ea73c288999d","Type":"ContainerStarted","Data":"1d607b10af1a317770b757a62d32c855808b3c529a0f3244c663c8fae66f98d0"} Oct 31 00:48:04.033898 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:04.033745 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" event={"ID":"e3616268-b555-45bb-9d19-ea73c288999d","Type":"ContainerStarted","Data":"09d6bd568f9cc7f5889cacf19c414bd11b21c7070f301e1dacd2b1e7f0245ad6"} Oct 31 00:48:04.148145 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:04.148086 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" podStartSLOduration=2.148072055 podStartE2EDuration="2.148072055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:04.145229115 +0000 UTC m=+1165.681371421" watchObservedRunningTime="2026-10-31 00:48:04.148072055 +0000 UTC m=+1165.684214359" Oct 31 00:48:04.277747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:04.277655 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:48:04.277747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:04.277731 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:48:05.040890 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:05.040854 5719 generic.go:356] "Generic (PLEG): container finished" podID="e3616268-b555-45bb-9d19-ea73c288999d" containerID="1d607b10af1a317770b757a62d32c855808b3c529a0f3244c663c8fae66f98d0" exitCode=0 Oct 31 00:48:05.041287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:05.040941 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" event={"ID":"e3616268-b555-45bb-9d19-ea73c288999d","Type":"ContainerDied","Data":"1d607b10af1a317770b757a62d32c855808b3c529a0f3244c663c8fae66f98d0"} Oct 31 00:48:05.277125 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:05.277091 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:48:05.277296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:05.277155 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:48:06.240501 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.240479 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:06.276802 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.276774 5719 patch_prober.go:29] interesting pod/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:48:06.276920 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.276826 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 00:48:06.362477 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.362449 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-openshift-ca\") pod \"e3616268-b555-45bb-9d19-ea73c288999d\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " Oct 31 00:48:06.362674 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.362567 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3616268-b555-45bb-9d19-ea73c288999d-kube-api-access-rt4bk\" (UniqueName: \"kubernetes.io/projected/e3616268-b555-45bb-9d19-ea73c288999d-kube-api-access-rt4bk\") pod \"e3616268-b555-45bb-9d19-ea73c288999d\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " Oct 31 00:48:06.362674 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.362668 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-xks-ca\") pod \"e3616268-b555-45bb-9d19-ea73c288999d\" (UID: \"e3616268-b555-45bb-9d19-ea73c288999d\") " Oct 31 00:48:06.362843 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.362813 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-openshift-ca" pod "e3616268-b555-45bb-9d19-ea73c288999d" (UID: "e3616268-b555-45bb-9d19-ea73c288999d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:06.363142 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.363124 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.365282 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.365255 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-xks-ca" pod "e3616268-b555-45bb-9d19-ea73c288999d" (UID: "e3616268-b555-45bb-9d19-ea73c288999d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:06.365395 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.365313 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3616268-b555-45bb-9d19-ea73c288999d-kube-api-access-rt4bk" pod "e3616268-b555-45bb-9d19-ea73c288999d" (UID: "e3616268-b555-45bb-9d19-ea73c288999d"). InnerVolumeSpecName "kube-api-access-rt4bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:06.464759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.464725 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4bk\" (UniqueName: \"kubernetes.io/projected/e3616268-b555-45bb-9d19-ea73c288999d-kube-api-access-rt4bk\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.464759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:06.464752 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3616268-b555-45bb-9d19-ea73c288999d-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:07.055031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:07.054964 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" Oct 31 00:48:07.055031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:07.054974 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf" event={"ID":"e3616268-b555-45bb-9d19-ea73c288999d","Type":"ContainerDied","Data":"09d6bd568f9cc7f5889cacf19c414bd11b21c7070f301e1dacd2b1e7f0245ad6"} Oct 31 00:48:07.055206 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:07.055032 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09d6bd568f9cc7f5889cacf19c414bd11b21c7070f301e1dacd2b1e7f0245ad6" Oct 31 00:48:07.318584 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:07.318559 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:07.319080 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:07.318936 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:07.319658 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:07.319640 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:48:18.142912 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.142833 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt"] Oct 31 00:48:18.143399 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.143383 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e3616268-b555-45bb-9d19-ea73c288999d" containerName="revoke-keys" Oct 31 00:48:18.143444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.143400 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3616268-b555-45bb-9d19-ea73c288999d" containerName="revoke-keys" Oct 31 00:48:18.143486 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.143475 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3616268-b555-45bb-9d19-ea73c288999d" containerName="revoke-keys" Oct 31 00:48:18.146834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.146818 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.198798 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.198768 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:18.246313 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.246285 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:18.246313 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.246299 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:18.246473 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.246443 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:48:18.249383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.249364 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt"] Oct 31 00:48:18.283997 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.283975 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlmt\" (UniqueName: \"kubernetes.io/projected/5c1fdda0-ebdd-43bd-abab-722c4010b1db-kube-api-access-ftlmt\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.284114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.284012 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.284199 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.284177 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.385171 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.385139 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlmt\" (UniqueName: \"kubernetes.io/projected/5c1fdda0-ebdd-43bd-abab-722c4010b1db-kube-api-access-ftlmt\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.385278 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.385188 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.385278 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.385263 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.385989 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.385926 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.387799 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.387774 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.444413 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.444359 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlmt\" (UniqueName: \"kubernetes.io/projected/5c1fdda0-ebdd-43bd-abab-722c4010b1db-kube-api-access-ftlmt\") pod \"maas-api-revoke-sub-02e2323d-sw7tt\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.459401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.459381 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:18.832381 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.832340 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt"] Oct 31 00:48:18.836979 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:18.836947 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1fdda0_ebdd_43bd_abab_722c4010b1db.slice/crio-22b353a4ccfc0c066f6391aeaaf6028f45716b9ac9020e3a06647a9571f86b36 WatchSource:0}: Error finding container 22b353a4ccfc0c066f6391aeaaf6028f45716b9ac9020e3a06647a9571f86b36: Status 404 returned error can't find the container with id 22b353a4ccfc0c066f6391aeaaf6028f45716b9ac9020e3a06647a9571f86b36 Oct 31 00:48:18.906120 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.906082 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8"] Oct 31 00:48:18.911895 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.911872 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.991668 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.991779 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.991820 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.991854 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8"] Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.991971 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.992020 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.992038 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz27s\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-kube-api-access-nz27s\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.992081 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9d5c6-bc61-4014-913e-e9a948da2594-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.992105 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:18.995734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:18.992149 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.092864 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.092784 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093018 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.092890 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093018 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.092938 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093018 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.092957 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz27s\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-kube-api-access-nz27s\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093199 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093140 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9d5c6-bc61-4014-913e-e9a948da2594-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093199 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093189 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093237 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093362 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093311 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093362 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093357 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093471 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093398 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093644 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093613 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093666 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093985 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093956 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.093985 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.093976 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9d5c6-bc61-4014-913e-e9a948da2594-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.095841 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.095813 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.095930 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.095884 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.129052 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.129023 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" event={"ID":"5c1fdda0-ebdd-43bd-abab-722c4010b1db","Type":"ContainerStarted","Data":"22b353a4ccfc0c066f6391aeaaf6028f45716b9ac9020e3a06647a9571f86b36"} Oct 31 00:48:19.152650 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.152630 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.152990 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.152742 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz27s\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-kube-api-access-nz27s\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.225253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.225232 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:19.258972 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.258932 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr"] Oct 31 00:48:19.263864 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.263847 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.320205 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.319226 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr"] Oct 31 00:48:19.396905 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.396829 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.396905 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.396889 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.397088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.397001 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs5m5\" (UniqueName: \"kubernetes.io/projected/f81e02a5-6d58-4372-8a0a-3a85d0919f45-kube-api-access-zs5m5\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.498031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.498006 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.498149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.498053 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.498149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.498087 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs5m5\" (UniqueName: \"kubernetes.io/projected/f81e02a5-6d58-4372-8a0a-3a85d0919f45-kube-api-access-zs5m5\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.498719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.498676 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.500453 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.500431 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.557012 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.556984 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs5m5\" (UniqueName: \"kubernetes.io/projected/f81e02a5-6d58-4372-8a0a-3a85d0919f45-kube-api-access-zs5m5\") pod \"maas-api-revoke-sub-79b92f53-lvldr\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.579931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.579907 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:19.601090 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.601070 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8"] Oct 31 00:48:19.605547 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:19.605511 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e9d5c6_bc61_4014_913e_e9a948da2594.slice/crio-ed7bd75be11d6929cd14ec279d1fb9e1292998acf340a3725bb86abeee3130a0 WatchSource:0}: Error finding container ed7bd75be11d6929cd14ec279d1fb9e1292998acf340a3725bb86abeee3130a0: Status 404 returned error can't find the container with id ed7bd75be11d6929cd14ec279d1fb9e1292998acf340a3725bb86abeee3130a0 Oct 31 00:48:19.608154 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.608110 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:19.608277 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.608187 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:19.608277 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.608215 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:19.956097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:19.956012 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr"] Oct 31 00:48:19.961680 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:19.961642 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81e02a5_6d58_4372_8a0a_3a85d0919f45.slice/crio-030b2bf453a83e776a58fbd4a812dfe41f428b3a857b9102397e1b377bdcc1ca WatchSource:0}: Error finding container 030b2bf453a83e776a58fbd4a812dfe41f428b3a857b9102397e1b377bdcc1ca: Status 404 returned error can't find the container with id 030b2bf453a83e776a58fbd4a812dfe41f428b3a857b9102397e1b377bdcc1ca Oct 31 00:48:20.136876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.136836 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" event={"ID":"b2e9d5c6-bc61-4014-913e-e9a948da2594","Type":"ContainerStarted","Data":"c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10"} Oct 31 00:48:20.137068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.136903 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" event={"ID":"b2e9d5c6-bc61-4014-913e-e9a948da2594","Type":"ContainerStarted","Data":"ed7bd75be11d6929cd14ec279d1fb9e1292998acf340a3725bb86abeee3130a0"} Oct 31 00:48:20.138185 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.138156 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" event={"ID":"f81e02a5-6d58-4372-8a0a-3a85d0919f45","Type":"ContainerStarted","Data":"030b2bf453a83e776a58fbd4a812dfe41f428b3a857b9102397e1b377bdcc1ca"} Oct 31 00:48:20.139674 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.139655 5719 generic.go:356] "Generic (PLEG): container finished" podID="5c1fdda0-ebdd-43bd-abab-722c4010b1db" containerID="b9b6887db39639f25e95618cbfc6bb8fbdcb6431b5b09a87679514b7420e4a61" exitCode=0 Oct 31 00:48:20.139768 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.139745 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" event={"ID":"5c1fdda0-ebdd-43bd-abab-722c4010b1db","Type":"ContainerDied","Data":"b9b6887db39639f25e95618cbfc6bb8fbdcb6431b5b09a87679514b7420e4a61"} Oct 31 00:48:20.225878 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.225536 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:20.227363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.227336 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:48:20.227479 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.227393 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:48:20.253265 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:20.253219 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podStartSLOduration=2.253205872 podStartE2EDuration="2.253205872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:20.252105402 +0000 UTC m=+1181.788247709" watchObservedRunningTime="2026-10-31 00:48:20.253205872 +0000 UTC m=+1181.789348177" Oct 31 00:48:21.147815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.147780 5719 generic.go:356] "Generic (PLEG): container finished" podID="f81e02a5-6d58-4372-8a0a-3a85d0919f45" containerID="9c3e5684e27f0d2488e9d2ebd03acf6a7be64ce795e627fa9c7548a85d52290e" exitCode=0 Oct 31 00:48:21.147981 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.147865 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" event={"ID":"f81e02a5-6d58-4372-8a0a-3a85d0919f45","Type":"ContainerDied","Data":"9c3e5684e27f0d2488e9d2ebd03acf6a7be64ce795e627fa9c7548a85d52290e"} Oct 31 00:48:21.226933 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.226468 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:48:21.226933 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.226526 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:48:21.376419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.376393 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:21.486586 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.486511 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml"] Oct 31 00:48:21.487067 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.487053 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5c1fdda0-ebdd-43bd-abab-722c4010b1db" containerName="revoke-keys" Oct 31 00:48:21.487123 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.487069 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1fdda0-ebdd-43bd-abab-722c4010b1db" containerName="revoke-keys" Oct 31 00:48:21.487160 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.487147 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1fdda0-ebdd-43bd-abab-722c4010b1db" containerName="revoke-keys" Oct 31 00:48:21.490396 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.490373 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.545566 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.545525 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1fdda0-ebdd-43bd-abab-722c4010b1db-kube-api-access-ftlmt\" (UniqueName: \"kubernetes.io/projected/5c1fdda0-ebdd-43bd-abab-722c4010b1db-kube-api-access-ftlmt\") pod \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " Oct 31 00:48:21.545762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.545665 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-openshift-ca\") pod \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " Oct 31 00:48:21.545865 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.545847 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-xks-ca\") pod \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\" (UID: \"5c1fdda0-ebdd-43bd-abab-722c4010b1db\") " Oct 31 00:48:21.547921 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.547890 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-openshift-ca" pod "5c1fdda0-ebdd-43bd-abab-722c4010b1db" (UID: "5c1fdda0-ebdd-43bd-abab-722c4010b1db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:21.548483 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.548459 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml"] Oct 31 00:48:21.548907 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.548882 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-xks-ca" pod "5c1fdda0-ebdd-43bd-abab-722c4010b1db" (UID: "5c1fdda0-ebdd-43bd-abab-722c4010b1db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:21.549182 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.549160 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1fdda0-ebdd-43bd-abab-722c4010b1db-kube-api-access-ftlmt" pod "5c1fdda0-ebdd-43bd-abab-722c4010b1db" (UID: "5c1fdda0-ebdd-43bd-abab-722c4010b1db"). InnerVolumeSpecName "kube-api-access-ftlmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:21.647410 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.647379 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.647557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.647537 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkh2\" (UniqueName: \"kubernetes.io/projected/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-kube-api-access-4gkh2\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.647610 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.647597 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.647765 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.647744 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:21.647765 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.647759 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c1fdda0-ebdd-43bd-abab-722c4010b1db-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:21.647848 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.647770 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftlmt\" (UniqueName: \"kubernetes.io/projected/5c1fdda0-ebdd-43bd-abab-722c4010b1db-kube-api-access-ftlmt\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:21.749021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.748949 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkh2\" (UniqueName: \"kubernetes.io/projected/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-kube-api-access-4gkh2\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.749021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.749006 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.749242 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.749102 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.749781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.749760 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.751457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.751438 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.825243 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.825210 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkh2\" (UniqueName: \"kubernetes.io/projected/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-kube-api-access-4gkh2\") pod \"maas-api-revoke-sub-07df7525-hlnml\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:21.832524 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:21.832498 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:22.059582 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.059497 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g"] Oct 31 00:48:22.062067 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.062044 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e229c3f2-g5k9g"] Oct 31 00:48:22.155266 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.155240 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" Oct 31 00:48:22.155448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.155236 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt" event={"ID":"5c1fdda0-ebdd-43bd-abab-722c4010b1db","Type":"ContainerDied","Data":"22b353a4ccfc0c066f6391aeaaf6028f45716b9ac9020e3a06647a9571f86b36"} Oct 31 00:48:22.155448 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.155370 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22b353a4ccfc0c066f6391aeaaf6028f45716b9ac9020e3a06647a9571f86b36" Oct 31 00:48:22.230874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.230851 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml"] Oct 31 00:48:22.231256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.231219 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:48:22.231321 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.231281 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:48:22.341286 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.341263 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:22.457636 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.457604 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-openshift-ca\") pod \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " Oct 31 00:48:22.457807 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.457788 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81e02a5-6d58-4372-8a0a-3a85d0919f45-kube-api-access-zs5m5\" (UniqueName: \"kubernetes.io/projected/f81e02a5-6d58-4372-8a0a-3a85d0919f45-kube-api-access-zs5m5\") pod \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " Oct 31 00:48:22.457922 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.457902 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-openshift-ca" pod "f81e02a5-6d58-4372-8a0a-3a85d0919f45" (UID: "f81e02a5-6d58-4372-8a0a-3a85d0919f45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:22.457962 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.457905 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-xks-ca\") pod \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\" (UID: \"f81e02a5-6d58-4372-8a0a-3a85d0919f45\") " Oct 31 00:48:22.458439 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.458414 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:22.460232 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.460200 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-xks-ca" pod "f81e02a5-6d58-4372-8a0a-3a85d0919f45" (UID: "f81e02a5-6d58-4372-8a0a-3a85d0919f45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:22.460477 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.460450 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81e02a5-6d58-4372-8a0a-3a85d0919f45-kube-api-access-zs5m5" pod "f81e02a5-6d58-4372-8a0a-3a85d0919f45" (UID: "f81e02a5-6d58-4372-8a0a-3a85d0919f45"). InnerVolumeSpecName "kube-api-access-zs5m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:22.560656 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.560617 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs5m5\" (UniqueName: \"kubernetes.io/projected/f81e02a5-6d58-4372-8a0a-3a85d0919f45-kube-api-access-zs5m5\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:22.560656 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.560650 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81e02a5-6d58-4372-8a0a-3a85d0919f45-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:22.625764 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:22.610432 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0586fec1-3b38-45af-a13b-e6acb464d2ec" path="/var/lib/kubelet/pods/0586fec1-3b38-45af-a13b-e6acb464d2ec/volumes" Oct 31 00:48:23.163318 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:23.163283 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" event={"ID":"b22c368a-fd32-4e28-bcb8-189a25a2e3ec","Type":"ContainerStarted","Data":"f5d835e456720c170166917d965f4d8b5385105f47bb658b734a2d8422c33910"} Oct 31 00:48:23.163579 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:23.163559 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" event={"ID":"b22c368a-fd32-4e28-bcb8-189a25a2e3ec","Type":"ContainerStarted","Data":"aabfabc40f0158a9ac3a68bd88bb02b42a2bc8e0130ceb2d7bde56e3695a943b"} Oct 31 00:48:23.164802 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:23.164785 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" event={"ID":"f81e02a5-6d58-4372-8a0a-3a85d0919f45","Type":"ContainerDied","Data":"030b2bf453a83e776a58fbd4a812dfe41f428b3a857b9102397e1b377bdcc1ca"} Oct 31 00:48:23.164919 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:23.164909 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="030b2bf453a83e776a58fbd4a812dfe41f428b3a857b9102397e1b377bdcc1ca" Oct 31 00:48:23.165004 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:23.164828 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr" Oct 31 00:48:23.225950 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:23.225919 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:48:23.226104 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:23.225966 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:48:23.287716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:23.287646 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" podStartSLOduration=2.287634146 podStartE2EDuration="2.287634146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:23.286222741 +0000 UTC m=+1184.822365047" watchObservedRunningTime="2026-10-31 00:48:23.287634146 +0000 UTC m=+1184.823776482" Oct 31 00:48:24.173728 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:24.172456 5719 generic.go:356] "Generic (PLEG): container finished" podID="b22c368a-fd32-4e28-bcb8-189a25a2e3ec" containerID="f5d835e456720c170166917d965f4d8b5385105f47bb658b734a2d8422c33910" exitCode=0 Oct 31 00:48:24.173728 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:24.172544 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" event={"ID":"b22c368a-fd32-4e28-bcb8-189a25a2e3ec","Type":"ContainerDied","Data":"f5d835e456720c170166917d965f4d8b5385105f47bb658b734a2d8422c33910"} Oct 31 00:48:24.225680 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:24.225647 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:48:24.225877 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:24.225733 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:48:24.872569 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:24.872525 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:48:25.226721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.226149 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:48:25.226721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.226207 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:48:25.378437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.378201 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:25.393848 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.393823 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-xks-ca\") pod \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " Oct 31 00:48:25.393985 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.393923 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-openshift-ca\") pod \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " Oct 31 00:48:25.393985 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.393957 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-kube-api-access-4gkh2\" (UniqueName: \"kubernetes.io/projected/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-kube-api-access-4gkh2\") pod \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\" (UID: \"b22c368a-fd32-4e28-bcb8-189a25a2e3ec\") " Oct 31 00:48:25.394905 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.394879 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-openshift-ca" pod "b22c368a-fd32-4e28-bcb8-189a25a2e3ec" (UID: "b22c368a-fd32-4e28-bcb8-189a25a2e3ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:25.396833 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.396788 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-kube-api-access-4gkh2" pod "b22c368a-fd32-4e28-bcb8-189a25a2e3ec" (UID: "b22c368a-fd32-4e28-bcb8-189a25a2e3ec"). InnerVolumeSpecName "kube-api-access-4gkh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:25.396833 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.396808 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-xks-ca" pod "b22c368a-fd32-4e28-bcb8-189a25a2e3ec" (UID: "b22c368a-fd32-4e28-bcb8-189a25a2e3ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:25.497416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.496239 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:25.497416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.496271 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:25.497416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:25.496288 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gkh2\" (UniqueName: \"kubernetes.io/projected/b22c368a-fd32-4e28-bcb8-189a25a2e3ec-kube-api-access-4gkh2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:26.220737 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:26.218811 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" Oct 31 00:48:26.224727 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:26.221738 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml" event={"ID":"b22c368a-fd32-4e28-bcb8-189a25a2e3ec","Type":"ContainerDied","Data":"aabfabc40f0158a9ac3a68bd88bb02b42a2bc8e0130ceb2d7bde56e3695a943b"} Oct 31 00:48:26.224727 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:26.221793 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aabfabc40f0158a9ac3a68bd88bb02b42a2bc8e0130ceb2d7bde56e3695a943b" Oct 31 00:48:26.236725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:26.233187 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:48:26.236725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:26.233241 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 00:48:27.182829 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.182773 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.294809 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.294765 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.295999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.295970 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.394718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.390774 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.394718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.392917 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 00:48:27.465075 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.464941 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:27.565724 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.561776 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.793389 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.789841 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.821713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.820815 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.897856 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.897782 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:27.963900 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.963850 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:27.964227 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:27.964197 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:28.240463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:28.240373 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:28.240719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:28.240439 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:29.281832 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.281797 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:48:29.282312 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.281854 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:29.395016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.394988 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:29.395378 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.395356 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:29.396062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.396043 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 00:48:29.962428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.962400 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt"] Oct 31 00:48:29.963004 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.962984 5719 cpu_manager.go:414] "containerMap: removing container" podUID="b22c368a-fd32-4e28-bcb8-189a25a2e3ec" containerName="revoke-keys" Oct 31 00:48:29.963004 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.963004 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22c368a-fd32-4e28-bcb8-189a25a2e3ec" containerName="revoke-keys" Oct 31 00:48:29.963149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.963039 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f81e02a5-6d58-4372-8a0a-3a85d0919f45" containerName="revoke-keys" Oct 31 00:48:29.963149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.963048 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81e02a5-6d58-4372-8a0a-3a85d0919f45" containerName="revoke-keys" Oct 31 00:48:29.963149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.963148 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22c368a-fd32-4e28-bcb8-189a25a2e3ec" containerName="revoke-keys" Oct 31 00:48:29.963239 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.963160 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81e02a5-6d58-4372-8a0a-3a85d0919f45" containerName="revoke-keys" Oct 31 00:48:29.966738 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:29.966724 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.018548 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.018518 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:30.018738 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.018618 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:48:30.020204 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.020183 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt"] Oct 31 00:48:30.045016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.044993 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:30.045156 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.045065 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:30.045392 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.045372 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps99d\" (UniqueName: \"kubernetes.io/projected/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-kube-api-access-ps99d\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.045495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.045477 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.045598 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.045583 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.147106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.147074 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.147270 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.147139 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps99d\" (UniqueName: \"kubernetes.io/projected/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-kube-api-access-ps99d\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.147270 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.147231 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.147686 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.147668 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.149595 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.149574 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.206896 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.206867 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps99d\" (UniqueName: \"kubernetes.io/projected/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-kube-api-access-ps99d\") pod \"maas-api-revoke-sub-135c2f71-sv6kt\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.279862 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.279804 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:30.657217 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:30.657186 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt"] Oct 31 00:48:30.661899 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:30.661871 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2eaf0d_d419_49ef_b25f_b30ac65745cf.slice/crio-cdc3dfd1a79358abcb4fe420502615572b22efc70012dfffce497981c0ad597c WatchSource:0}: Error finding container cdc3dfd1a79358abcb4fe420502615572b22efc70012dfffce497981c0ad597c: Status 404 returned error can't find the container with id cdc3dfd1a79358abcb4fe420502615572b22efc70012dfffce497981c0ad597c Oct 31 00:48:31.276613 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:31.276525 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" event={"ID":"1b2eaf0d-d419-49ef-b25f-b30ac65745cf","Type":"ContainerStarted","Data":"57928598a04364d4082cccdfaa29034a14eb97c73625fe428f84f576a13688b6"} Oct 31 00:48:31.276613 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:31.276590 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" event={"ID":"1b2eaf0d-d419-49ef-b25f-b30ac65745cf","Type":"ContainerStarted","Data":"cdc3dfd1a79358abcb4fe420502615572b22efc70012dfffce497981c0ad597c"} Oct 31 00:48:31.390451 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:31.390402 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" podStartSLOduration=2.390388743 podStartE2EDuration="2.390388743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:31.389593558 +0000 UTC m=+1192.925735873" watchObservedRunningTime="2026-10-31 00:48:31.390388743 +0000 UTC m=+1192.926531047" Oct 31 00:48:32.284303 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.284270 5719 generic.go:356] "Generic (PLEG): container finished" podID="1b2eaf0d-d419-49ef-b25f-b30ac65745cf" containerID="57928598a04364d4082cccdfaa29034a14eb97c73625fe428f84f576a13688b6" exitCode=0 Oct 31 00:48:32.284675 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.284351 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" event={"ID":"1b2eaf0d-d419-49ef-b25f-b30ac65745cf","Type":"ContainerDied","Data":"57928598a04364d4082cccdfaa29034a14eb97c73625fe428f84f576a13688b6"} Oct 31 00:48:32.674796 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.674765 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq"] Oct 31 00:48:32.678802 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.678779 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.735323 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.735290 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq"] Oct 31 00:48:32.778329 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.778295 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01760557-408a-4298-833c-6b948b61c428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.778522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.778460 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqcn9\" (UniqueName: \"kubernetes.io/projected/01760557-408a-4298-833c-6b948b61c428-kube-api-access-mqcn9\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.778600 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.778577 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01760557-408a-4298-833c-6b948b61c428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.880011 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.879980 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqcn9\" (UniqueName: \"kubernetes.io/projected/01760557-408a-4298-833c-6b948b61c428-kube-api-access-mqcn9\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.880221 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.880087 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01760557-408a-4298-833c-6b948b61c428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.880221 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.880153 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01760557-408a-4298-833c-6b948b61c428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.880779 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.880760 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01760557-408a-4298-833c-6b948b61c428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.882534 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.882514 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01760557-408a-4298-833c-6b948b61c428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:32.938968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:32.938902 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqcn9\" (UniqueName: \"kubernetes.io/projected/01760557-408a-4298-833c-6b948b61c428-kube-api-access-mqcn9\") pod \"maas-api-revoke-sub-726b07a7-b4fsq\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:33.001793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.001768 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:33.010048 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:48:33.010016 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:48:33.379906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.379877 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq"] Oct 31 00:48:33.434391 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.434370 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:33.487045 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.487024 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-kube-api-access-ps99d\" (UniqueName: \"kubernetes.io/projected/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-kube-api-access-ps99d\") pod \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " Oct 31 00:48:33.487154 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.487141 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-xks-ca\") pod \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " Oct 31 00:48:33.487214 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.487176 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-openshift-ca\") pod \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\" (UID: \"1b2eaf0d-d419-49ef-b25f-b30ac65745cf\") " Oct 31 00:48:33.487573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.487543 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-openshift-ca" pod "1b2eaf0d-d419-49ef-b25f-b30ac65745cf" (UID: "1b2eaf0d-d419-49ef-b25f-b30ac65745cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:33.487858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.487648 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:33.489146 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.489114 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-xks-ca" pod "1b2eaf0d-d419-49ef-b25f-b30ac65745cf" (UID: "1b2eaf0d-d419-49ef-b25f-b30ac65745cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:33.489235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.489192 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-kube-api-access-ps99d" pod "1b2eaf0d-d419-49ef-b25f-b30ac65745cf" (UID: "1b2eaf0d-d419-49ef-b25f-b30ac65745cf"). InnerVolumeSpecName "kube-api-access-ps99d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:33.589032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.589007 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:33.589032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:33.589031 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps99d\" (UniqueName: \"kubernetes.io/projected/1b2eaf0d-d419-49ef-b25f-b30ac65745cf-kube-api-access-ps99d\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:34.301041 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:34.301015 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" Oct 31 00:48:34.301241 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:34.301014 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt" event={"ID":"1b2eaf0d-d419-49ef-b25f-b30ac65745cf","Type":"ContainerDied","Data":"cdc3dfd1a79358abcb4fe420502615572b22efc70012dfffce497981c0ad597c"} Oct 31 00:48:34.301241 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:34.301132 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdc3dfd1a79358abcb4fe420502615572b22efc70012dfffce497981c0ad597c" Oct 31 00:48:34.302951 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:34.302926 5719 generic.go:356] "Generic (PLEG): container finished" podID="01760557-408a-4298-833c-6b948b61c428" containerID="af0b1d1e8c2505bb1f50ec3735108a30d790cb10d9c9f5eb3eba2316d2615097" exitCode=0 Oct 31 00:48:34.303093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:34.303006 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" event={"ID":"01760557-408a-4298-833c-6b948b61c428","Type":"ContainerDied","Data":"af0b1d1e8c2505bb1f50ec3735108a30d790cb10d9c9f5eb3eba2316d2615097"} Oct 31 00:48:34.303093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:34.303053 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" event={"ID":"01760557-408a-4298-833c-6b948b61c428","Type":"ContainerStarted","Data":"29b3f50f7cd92f7f135e9ca3d08f98a2d7961e250426b4e7711bce3938356322"} Oct 31 00:48:35.456164 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.456134 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:35.509531 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.509509 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01760557-408a-4298-833c-6b948b61c428-kube-api-access-mqcn9\" (UniqueName: \"kubernetes.io/projected/01760557-408a-4298-833c-6b948b61c428-kube-api-access-mqcn9\") pod \"01760557-408a-4298-833c-6b948b61c428\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " Oct 31 00:48:35.509663 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.509642 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01760557-408a-4298-833c-6b948b61c428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01760557-408a-4298-833c-6b948b61c428-maas-api-openshift-ca\") pod \"01760557-408a-4298-833c-6b948b61c428\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " Oct 31 00:48:35.509765 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.509721 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01760557-408a-4298-833c-6b948b61c428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01760557-408a-4298-833c-6b948b61c428-maas-api-xks-ca\") pod \"01760557-408a-4298-833c-6b948b61c428\" (UID: \"01760557-408a-4298-833c-6b948b61c428\") " Oct 31 00:48:35.509981 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.509958 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01760557-408a-4298-833c-6b948b61c428-maas-api-openshift-ca" pod "01760557-408a-4298-833c-6b948b61c428" (UID: "01760557-408a-4298-833c-6b948b61c428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:35.510119 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.510101 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01760557-408a-4298-833c-6b948b61c428-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:35.511803 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.511775 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01760557-408a-4298-833c-6b948b61c428-kube-api-access-mqcn9" pod "01760557-408a-4298-833c-6b948b61c428" (UID: "01760557-408a-4298-833c-6b948b61c428"). InnerVolumeSpecName "kube-api-access-mqcn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:35.511939 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.511922 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01760557-408a-4298-833c-6b948b61c428-maas-api-xks-ca" pod "01760557-408a-4298-833c-6b948b61c428" (UID: "01760557-408a-4298-833c-6b948b61c428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:35.610835 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.610811 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01760557-408a-4298-833c-6b948b61c428-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:35.610835 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:35.610834 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqcn9\" (UniqueName: \"kubernetes.io/projected/01760557-408a-4298-833c-6b948b61c428-kube-api-access-mqcn9\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:36.317225 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.317196 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" event={"ID":"01760557-408a-4298-833c-6b948b61c428","Type":"ContainerDied","Data":"29b3f50f7cd92f7f135e9ca3d08f98a2d7961e250426b4e7711bce3938356322"} Oct 31 00:48:36.317401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.317235 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq" Oct 31 00:48:36.317401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.317243 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29b3f50f7cd92f7f135e9ca3d08f98a2d7961e250426b4e7711bce3938356322" Oct 31 00:48:36.849612 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.849584 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd"] Oct 31 00:48:36.850078 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.850063 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1b2eaf0d-d419-49ef-b25f-b30ac65745cf" containerName="revoke-keys" Oct 31 00:48:36.850078 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.850077 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2eaf0d-d419-49ef-b25f-b30ac65745cf" containerName="revoke-keys" Oct 31 00:48:36.850184 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.850088 5719 cpu_manager.go:414] "containerMap: removing container" podUID="01760557-408a-4298-833c-6b948b61c428" containerName="revoke-keys" Oct 31 00:48:36.850184 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.850093 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01760557-408a-4298-833c-6b948b61c428" containerName="revoke-keys" Oct 31 00:48:36.850252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.850192 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2eaf0d-d419-49ef-b25f-b30ac65745cf" containerName="revoke-keys" Oct 31 00:48:36.850252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.850203 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="01760557-408a-4298-833c-6b948b61c428" containerName="revoke-keys" Oct 31 00:48:36.858715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.853982 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.886756 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.886730 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw"] Oct 31 00:48:36.890537 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.890519 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:36.913105 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.913081 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd"] Oct 31 00:48:36.923588 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.923569 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.923676 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.923600 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.923766 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.923752 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b09fe53-f6c6-4487-9470-68d796bdf847-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.923806 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.923798 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.923877 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.923861 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.923944 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.923903 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.923977 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.923959 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.924017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.924006 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:36.924055 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:36.924025 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zzr\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-kube-api-access-v2zzr\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.005686 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.005649 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw"] Oct 31 00:48:37.025220 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025189 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.025359 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025237 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.025359 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025270 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.025359 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025327 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.025536 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025516 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.025596 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025563 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b09fe53-f6c6-4487-9470-68d796bdf847-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.025596 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025581 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.025723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025633 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.025723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025655 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.025824 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025757 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.025824 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025806 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.025936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025862 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.025936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025914 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.026042 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025957 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.026042 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.025984 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.026144 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.026041 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.026144 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.026067 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx9v7\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-kube-api-access-kx9v7\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.026144 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.026112 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.026144 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.026135 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.026358 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.026071 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b09fe53-f6c6-4487-9470-68d796bdf847-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.026358 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.026273 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zzr\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-kube-api-access-v2zzr\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.026467 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.026423 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.026712 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.026671 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.028243 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.028224 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.028721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.028681 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.084839 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.084808 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.085043 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.085026 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zzr\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-kube-api-access-v2zzr\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.128376 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128325 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128376 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128364 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128505 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128481 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128549 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128536 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128595 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128583 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128636 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128622 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx9v7\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-kube-api-access-kx9v7\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128712 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128681 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128764 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128739 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-data\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128764 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128754 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128873 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128831 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.128918 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.128903 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-certs\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.129058 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.129036 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.129188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.129169 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-credential-socket\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.129188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.129182 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istiod-ca-cert\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.130712 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.130676 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-envoy\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.131146 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.131128 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-podinfo\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.170443 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.170421 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:37.187209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.187184 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-token\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.187308 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.187293 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx9v7\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-kube-api-access-kx9v7\") pod \"e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.204098 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.204075 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:37.600385 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.600360 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd"] Oct 31 00:48:37.603835 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.603812 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw"] Oct 31 00:48:37.604820 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:37.604797 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b09fe53_f6c6_4487_9470_68d796bdf847.slice/crio-6aebd35430fb6d9225b0775e5f98c50d9c74d364e1f201d204dcde2c3313d9d2 WatchSource:0}: Error finding container 6aebd35430fb6d9225b0775e5f98c50d9c74d364e1f201d204dcde2c3313d9d2: Status 404 returned error can't find the container with id 6aebd35430fb6d9225b0775e5f98c50d9c74d364e1f201d204dcde2c3313d9d2 Oct 31 00:48:37.606975 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:37.606950 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6818ade5_3ff1_4f90_a9f8_53acf2132d51.slice/crio-2839764a0c0faea2373230d82d4ccb293a935185eadd8b1da0c6af65b522fc86 WatchSource:0}: Error finding container 2839764a0c0faea2373230d82d4ccb293a935185eadd8b1da0c6af65b522fc86: Status 404 returned error can't find the container with id 2839764a0c0faea2373230d82d4ccb293a935185eadd8b1da0c6af65b522fc86 Oct 31 00:48:37.607616 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.607583 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:37.607729 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.607654 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:37.607729 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.607718 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:37.608973 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.608949 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:37.609068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.609014 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:37.609131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:37.609080 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:38.363218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.363182 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" event={"ID":"6818ade5-3ff1-4f90-a9f8-53acf2132d51","Type":"ContainerStarted","Data":"2e0e8fd7278a39215c6784621a46a364722c93bd055c8b99f7d03580ad33f481"} Oct 31 00:48:38.363742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.363244 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" event={"ID":"6818ade5-3ff1-4f90-a9f8-53acf2132d51","Type":"ContainerStarted","Data":"2839764a0c0faea2373230d82d4ccb293a935185eadd8b1da0c6af65b522fc86"} Oct 31 00:48:38.365403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.365378 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" event={"ID":"1b09fe53-f6c6-4487-9470-68d796bdf847","Type":"ContainerStarted","Data":"a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e"} Oct 31 00:48:38.365469 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.365424 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" event={"ID":"1b09fe53-f6c6-4487-9470-68d796bdf847","Type":"ContainerStarted","Data":"6aebd35430fb6d9225b0775e5f98c50d9c74d364e1f201d204dcde2c3313d9d2"} Oct 31 00:48:38.480471 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.480423 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podStartSLOduration=2.480409329 podStartE2EDuration="2.480409329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:38.480060932 +0000 UTC m=+1200.016203236" watchObservedRunningTime="2026-10-31 00:48:38.480409329 +0000 UTC m=+1200.016551639" Oct 31 00:48:38.620780 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.610625 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podStartSLOduration=2.610603265 podStartE2EDuration="2.610603265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:38.599093509 +0000 UTC m=+1200.135235811" watchObservedRunningTime="2026-10-31 00:48:38.610603265 +0000 UTC m=+1200.146745571" Oct 31 00:48:38.781828 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.781796 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b"] Oct 31 00:48:38.786793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.786770 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:38.855755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.847383 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b"] Oct 31 00:48:38.855755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.848349 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:48:38.855755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.848407 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:38.855755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.848464 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:38.855755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.848538 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:38.867599 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.867564 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:38.867782 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.867756 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:38.867867 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.867836 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfhz\" (UniqueName: \"kubernetes.io/projected/ead3b38f-166b-4092-b6f8-5d4418411b3a-kube-api-access-2wfhz\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:38.970127 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.970042 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfhz\" (UniqueName: \"kubernetes.io/projected/ead3b38f-166b-4092-b6f8-5d4418411b3a-kube-api-access-2wfhz\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:38.970277 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.970207 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:38.970357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.970341 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:38.971012 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.970991 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:38.973044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:38.973021 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:39.055277 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.043755 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfhz\" (UniqueName: \"kubernetes.io/projected/ead3b38f-166b-4092-b6f8-5d4418411b3a-kube-api-access-2wfhz\") pod \"maas-api-revoke-sub-987dcff7-bnf2b\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:39.127595 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.127462 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:39.170716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.170662 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:39.171836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.171809 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:48:39.171954 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.171860 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:48:39.214726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.210328 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:39.214726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.210497 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:48:39.214726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.210542 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:48:39.558530 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:39.558445 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b"] Oct 31 00:48:40.054212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.054142 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s"] Oct 31 00:48:40.057536 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.057510 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d55191-7sf2s"] Oct 31 00:48:40.171116 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.171091 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:48:40.171241 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.171136 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:48:40.219714 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.216014 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:48:40.219714 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.216067 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:48:40.380674 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.380635 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" event={"ID":"ead3b38f-166b-4092-b6f8-5d4418411b3a","Type":"ContainerStarted","Data":"5635bdab22f5a547512d9a0b19e17b49d5287261472c58a3397b4b65640929d1"} Oct 31 00:48:40.380892 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.380712 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" event={"ID":"ead3b38f-166b-4092-b6f8-5d4418411b3a","Type":"ContainerStarted","Data":"fb62209ff9f0904315d7d858b956294ef0752ee24bd13a633679ba3c5a62eeee"} Oct 31 00:48:40.533769 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.530959 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" podStartSLOduration=2.530940492 podStartE2EDuration="2.530940492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:40.529992064 +0000 UTC m=+1202.066134369" watchObservedRunningTime="2026-10-31 00:48:40.530940492 +0000 UTC m=+1202.067082798" Oct 31 00:48:40.630713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:40.630506 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a5ee03-d538-46ac-a24b-2a4bba48e80d" path="/var/lib/kubelet/pods/32a5ee03-d538-46ac-a24b-2a4bba48e80d/volumes" Oct 31 00:48:41.172038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:41.171578 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:48:41.172038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:41.171621 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:48:41.233720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:41.231297 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:48:41.233720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:41.231353 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:48:41.388795 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:41.388761 5719 generic.go:356] "Generic (PLEG): container finished" podID="ead3b38f-166b-4092-b6f8-5d4418411b3a" containerID="5635bdab22f5a547512d9a0b19e17b49d5287261472c58a3397b4b65640929d1" exitCode=0 Oct 31 00:48:41.388964 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:41.388834 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" event={"ID":"ead3b38f-166b-4092-b6f8-5d4418411b3a","Type":"ContainerDied","Data":"5635bdab22f5a547512d9a0b19e17b49d5287261472c58a3397b4b65640929d1"} Oct 31 00:48:42.171585 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.171550 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:48:42.172009 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.171613 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:48:42.209093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.209048 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:48:42.209316 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.209292 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:48:42.680417 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.680382 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:42.760061 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.759979 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-openshift-ca\") pod \"ead3b38f-166b-4092-b6f8-5d4418411b3a\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " Oct 31 00:48:42.760061 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.760029 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-xks-ca\") pod \"ead3b38f-166b-4092-b6f8-5d4418411b3a\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " Oct 31 00:48:42.760286 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.760116 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead3b38f-166b-4092-b6f8-5d4418411b3a-kube-api-access-2wfhz\" (UniqueName: \"kubernetes.io/projected/ead3b38f-166b-4092-b6f8-5d4418411b3a-kube-api-access-2wfhz\") pod \"ead3b38f-166b-4092-b6f8-5d4418411b3a\" (UID: \"ead3b38f-166b-4092-b6f8-5d4418411b3a\") " Oct 31 00:48:42.761114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.761093 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-openshift-ca" pod "ead3b38f-166b-4092-b6f8-5d4418411b3a" (UID: "ead3b38f-166b-4092-b6f8-5d4418411b3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:42.763208 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.762592 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-xks-ca" pod "ead3b38f-166b-4092-b6f8-5d4418411b3a" (UID: "ead3b38f-166b-4092-b6f8-5d4418411b3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:42.763208 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.762630 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead3b38f-166b-4092-b6f8-5d4418411b3a-kube-api-access-2wfhz" pod "ead3b38f-166b-4092-b6f8-5d4418411b3a" (UID: "ead3b38f-166b-4092-b6f8-5d4418411b3a"). InnerVolumeSpecName "kube-api-access-2wfhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:42.861642 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.861605 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:42.861642 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.861634 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead3b38f-166b-4092-b6f8-5d4418411b3a-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:42.861642 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:42.861645 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfhz\" (UniqueName: \"kubernetes.io/projected/ead3b38f-166b-4092-b6f8-5d4418411b3a-kube-api-access-2wfhz\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:43.170885 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:43.170852 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:48:43.171095 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:43.170902 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:48:43.218718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:43.214038 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:48:43.218718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:43.214093 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:48:43.456535 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:43.456459 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" event={"ID":"ead3b38f-166b-4092-b6f8-5d4418411b3a","Type":"ContainerDied","Data":"fb62209ff9f0904315d7d858b956294ef0752ee24bd13a633679ba3c5a62eeee"} Oct 31 00:48:43.456765 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:43.456751 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb62209ff9f0904315d7d858b956294ef0752ee24bd13a633679ba3c5a62eeee" Oct 31 00:48:43.456894 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:43.456885 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b" Oct 31 00:48:44.171368 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:44.171334 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:48:44.171534 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:44.171408 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 00:48:44.204642 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:44.204606 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:48:44.204821 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:44.204678 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 00:48:45.028766 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.028108 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6"] Oct 31 00:48:45.029160 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.028987 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ead3b38f-166b-4092-b6f8-5d4418411b3a" containerName="revoke-keys" Oct 31 00:48:45.029160 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.029008 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead3b38f-166b-4092-b6f8-5d4418411b3a" containerName="revoke-keys" Oct 31 00:48:45.032752 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.029246 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead3b38f-166b-4092-b6f8-5d4418411b3a" containerName="revoke-keys" Oct 31 00:48:45.033686 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.033663 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.118280 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.118245 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6"] Oct 31 00:48:45.126356 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126322 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.126570 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126546 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.126652 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126599 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.126652 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126641 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.126817 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126671 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.126817 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126725 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.126817 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126790 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.126960 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126815 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b72ba3-cf95-40c0-a436-31921b9a72d9-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.126960 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.126948 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwt9\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-kube-api-access-xpwt9\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.176931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.176904 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:45.177418 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.177402 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:45.178295 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.178272 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:48:45.209751 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.209718 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:45.210026 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.210007 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:45.210730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.210683 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 00:48:45.228297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228267 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228434 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228307 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228434 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228357 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228434 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228381 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b72ba3-cf95-40c0-a436-31921b9a72d9-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228604 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228550 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwt9\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-kube-api-access-xpwt9\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228819 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228794 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228846 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228889 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228895 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.228979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228956 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.229248 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.229229 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.229295 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.228795 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.229394 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.229368 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b72ba3-cf95-40c0-a436-31921b9a72d9-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.229513 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.229493 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.231607 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.231581 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.231865 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.231849 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.296196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.288050 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwt9\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-kube-api-access-xpwt9\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.296196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.288967 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.351382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.351351 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:45.781220 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.781190 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6"] Oct 31 00:48:45.785000 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:45.784977 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b72ba3_cf95_40c0_a436_31921b9a72d9.slice/crio-07a8cea45a8e18ad478cb9be318cae67813831c8b86585396ef67575c496e14c WatchSource:0}: Error finding container 07a8cea45a8e18ad478cb9be318cae67813831c8b86585396ef67575c496e14c: Status 404 returned error can't find the container with id 07a8cea45a8e18ad478cb9be318cae67813831c8b86585396ef67575c496e14c Oct 31 00:48:45.790774 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.790740 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:45.790878 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.790807 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:45.790878 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:45.790837 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:46.051136 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:46.051044 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn"] Oct 31 00:48:46.055628 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:46.055604 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1015d04-8xpwn"] Oct 31 00:48:46.481757 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:46.481721 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" event={"ID":"34b72ba3-cf95-40c0-a436-31921b9a72d9","Type":"ContainerStarted","Data":"cca709f1b8f75da9891b8505c07fee59f01bdd077e7fdc51cb3fda74c84ec2a3"} Oct 31 00:48:46.481929 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:46.481809 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" event={"ID":"34b72ba3-cf95-40c0-a436-31921b9a72d9","Type":"ContainerStarted","Data":"07a8cea45a8e18ad478cb9be318cae67813831c8b86585396ef67575c496e14c"} Oct 31 00:48:46.609723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:46.599778 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podStartSLOduration=2.599761479 podStartE2EDuration="2.599761479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:46.598504132 +0000 UTC m=+1208.134646438" watchObservedRunningTime="2026-10-31 00:48:46.599761479 +0000 UTC m=+1208.135903786" Oct 31 00:48:46.628003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:46.627962 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71e60b3-25b0-4825-a86f-12648ab061fe" path="/var/lib/kubelet/pods/e71e60b3-25b0-4825-a86f-12648ab061fe/volumes" Oct 31 00:48:47.355403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.355362 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:47.357449 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.357408 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:47.357607 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.357495 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:47.595876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.595843 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d"] Oct 31 00:48:47.601943 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.601916 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.657910 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.657559 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d"] Oct 31 00:48:47.658477 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.658443 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.658596 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.658572 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.658661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.658617 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtc4k\" (UniqueName: \"kubernetes.io/projected/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-kube-api-access-xtc4k\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.660001 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.659978 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:48:47.723719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.723010 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:47.723719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.723718 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:47.723956 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.723789 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:47.759600 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.759567 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.759771 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.759737 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.759836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.759791 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtc4k\" (UniqueName: \"kubernetes.io/projected/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-kube-api-access-xtc4k\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.760403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.760375 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.762482 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.762462 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.827632 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.827602 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtc4k\" (UniqueName: \"kubernetes.io/projected/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-kube-api-access-xtc4k\") pod \"maas-api-revoke-sub-1ab2c482-cb44d\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:47.947513 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:47.947443 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:48.336198 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:48.336167 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d"] Oct 31 00:48:48.340950 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:48.340912 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5b44d5_caa1_44b4_bbe9_e77438cbd43c.slice/crio-9bb0c1ec5cf9f5419f44778ca57f0ed62633360367fe06c6d2a36ee93e0d06b1 WatchSource:0}: Error finding container 9bb0c1ec5cf9f5419f44778ca57f0ed62633360367fe06c6d2a36ee93e0d06b1: Status 404 returned error can't find the container with id 9bb0c1ec5cf9f5419f44778ca57f0ed62633360367fe06c6d2a36ee93e0d06b1 Oct 31 00:48:48.352164 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:48.352139 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:48.352266 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:48.352182 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:48.496523 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:48.496493 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" event={"ID":"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c","Type":"ContainerStarted","Data":"9bb0c1ec5cf9f5419f44778ca57f0ed62633360367fe06c6d2a36ee93e0d06b1"} Oct 31 00:48:49.142614 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.142586 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng"] Oct 31 00:48:49.146436 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.146421 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.177992 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178164 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.178026 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178164 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.178110 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178282 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.178172 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178282 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.178237 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178282 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.178260 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.178324 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.178382 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.178537 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.178427 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nhtw\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-kube-api-access-2nhtw\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.279608 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.279574 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.279608 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.279615 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.279836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.279746 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.279884 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.279865 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.279929 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.279897 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nhtw\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-kube-api-access-2nhtw\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.279965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.279928 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.280003 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.279969 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.280055 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.280006 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-data\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.280055 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.280040 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.280055 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.280043 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.280208 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.280094 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-credential-socket\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.280208 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.280155 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.280483 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.280465 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-certs\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.280766 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.280584 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istiod-ca-cert\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.282245 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.282224 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-envoy\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.282425 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.282408 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-podinfo\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.339783 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.339490 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-token\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.339783 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.339669 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nhtw\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-kube-api-access-2nhtw\") pod \"e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.342352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.342283 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng"] Oct 31 00:48:49.352562 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.352541 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:49.352683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.352590 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:49.460140 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.460111 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:49.529932 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.529895 5719 generic.go:356] "Generic (PLEG): container finished" podID="1c5b44d5-caa1-44b4-bbe9-e77438cbd43c" containerID="a2644db996590c0a4e46f73ee7bb611b68f396fb6aef3bb9d5a3e295fac35d32" exitCode=0 Oct 31 00:48:49.530457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.529970 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" event={"ID":"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c","Type":"ContainerDied","Data":"a2644db996590c0a4e46f73ee7bb611b68f396fb6aef3bb9d5a3e295fac35d32"} Oct 31 00:48:49.929716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.922332 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng"] Oct 31 00:48:49.981021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.975731 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:49.981021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.975824 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:49.981021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:49.975870 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:50.357716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:50.357099 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:50.357716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:50.357161 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:50.573028 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:50.572216 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" event={"ID":"3468d8a4-e84e-4c45-8e70-6433e0d42da6","Type":"ContainerStarted","Data":"990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383"} Oct 31 00:48:50.573028 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:50.572283 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" event={"ID":"3468d8a4-e84e-4c45-8e70-6433e0d42da6","Type":"ContainerStarted","Data":"75376698706b4da7850c4e832a1783d6c4fd5cdd9ba8c95e7f9d7b14713e53c4"} Oct 31 00:48:51.490720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.436638 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:51.490720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.452024 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:51.490720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.452078 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:51.499056 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.498968 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:48:51.572616 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.564388 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:51.585718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.579724 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:51.747719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.722422 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:51.748051 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.748017 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:51.827775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.820015 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:51.909760 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:51.902005 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.056729 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.055917 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:52.111775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.111440 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-openshift-ca\") pod \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " Oct 31 00:48:52.111775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.111623 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-kube-api-access-xtc4k\" (UniqueName: \"kubernetes.io/projected/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-kube-api-access-xtc4k\") pod \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " Oct 31 00:48:52.111775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.111715 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-xks-ca\") pod \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\" (UID: \"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c\") " Oct 31 00:48:52.157923 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.144242 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-openshift-ca" pod "1c5b44d5-caa1-44b4-bbe9-e77438cbd43c" (UID: "1c5b44d5-caa1-44b4-bbe9-e77438cbd43c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:52.173175 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.173136 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.186717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.184180 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-kube-api-access-xtc4k" pod "1c5b44d5-caa1-44b4-bbe9-e77438cbd43c" (UID: "1c5b44d5-caa1-44b4-bbe9-e77438cbd43c"). InnerVolumeSpecName "kube-api-access-xtc4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:52.186717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.184327 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-xks-ca" pod "1c5b44d5-caa1-44b4-bbe9-e77438cbd43c" (UID: "1c5b44d5-caa1-44b4-bbe9-e77438cbd43c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:52.235716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.221143 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podStartSLOduration=3.22112261 podStartE2EDuration="3.22112261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:50.778034637 +0000 UTC m=+1212.314176943" watchObservedRunningTime="2026-10-31 00:48:52.22112261 +0000 UTC m=+1213.757264919" Oct 31 00:48:52.295830 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.295780 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 31 00:48:52.307721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.303169 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:52.307721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.303202 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtc4k\" (UniqueName: \"kubernetes.io/projected/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-kube-api-access-xtc4k\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:52.307721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.303218 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:52.356719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.354776 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:52.356719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.354833 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:52.356719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.354931 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.454777 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.447726 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.454777 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.448561 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:48:52.479716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.457782 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:48:52.484503 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.484433 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:52.484503 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.484486 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:52.516718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.507530 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.591717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.581899 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 31 00:48:52.616719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.604749 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:48:52.641720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.616994 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.734772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.734366 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:48:52.900719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.899767 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 31 00:48:52.900719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.900261 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:48:52.912718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.912435 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:52.961716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:52.955764 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:48:53.479717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.449273 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:53.479717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.466840 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:48:53.499733 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.480828 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:48:53.499733 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.481623 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:53.499733 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.481682 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:53.499733 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.483552 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:53.499733 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.483590 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:53.547894 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.543965 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:53.547894 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.544331 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:48:53.547894 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.544632 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:48:53.547894 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.544709 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:48:53.582736 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.571040 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:48:53.583511 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.583175 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:53.606367 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.606337 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" Oct 31 00:48:53.611778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.611748 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d" event={"ID":"1c5b44d5-caa1-44b4-bbe9-e77438cbd43c","Type":"ContainerDied","Data":"9bb0c1ec5cf9f5419f44778ca57f0ed62633360367fe06c6d2a36ee93e0d06b1"} Oct 31 00:48:53.612038 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.612016 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bb0c1ec5cf9f5419f44778ca57f0ed62633360367fe06c6d2a36ee93e0d06b1" Oct 31 00:48:53.648057 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.648015 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:53.992806 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.990293 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 31 00:48:53.992806 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:53.991835 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Oct 31 00:48:54.417778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.354811 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:54.417778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.354862 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:54.453790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.441270 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:48:54.453790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.452796 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:54.491741 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.481654 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:54.491741 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.482578 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:54.491741 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.482617 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:54.563662 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.563598 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:54.564272 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.564238 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:54.592344 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.592306 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:54.594132 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.594100 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:54.700103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.700007 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.001824 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:54.991048 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.002120 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.002089 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.296683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.295561 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:48:55.352049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.352014 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:48:55.352326 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.352298 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 00:48:55.355791 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.355762 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.391082 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.390364 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:48:55.460168 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.460129 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.460671 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.460644 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 00:48:55.461088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.461070 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:48:55.520741 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.520683 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:48:55.521362 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.521337 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:55.521535 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.521512 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:55.603725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.599801 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:48:55.603725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.600140 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:48:55.623731 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.604936 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:48:55.718973 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.718943 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:48:55.785714 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.779725 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:48:55.789012 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.787458 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:48:55.791288 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:55.791263 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:48:56.460502 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:56.460469 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:56.460737 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:56.460520 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:56.531113 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:56.531081 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:56.531768 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:56.531740 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:56.532723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:56.532683 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:48:57.460642 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:57.460607 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:57.461103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:57.460662 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:58.048570 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.048532 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s"] Oct 31 00:48:58.095913 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.095882 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72dd6170-8td4s"] Oct 31 00:48:58.372017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.371985 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw"] Oct 31 00:48:58.372536 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.372523 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1c5b44d5-caa1-44b4-bbe9-e77438cbd43c" containerName="revoke-keys" Oct 31 00:48:58.372575 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.372537 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5b44d5-caa1-44b4-bbe9-e77438cbd43c" containerName="revoke-keys" Oct 31 00:48:58.372621 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.372617 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5b44d5-caa1-44b4-bbe9-e77438cbd43c" containerName="revoke-keys" Oct 31 00:48:58.376173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.376155 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.428043 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.428008 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:48:58.428212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.428064 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:48:58.428212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.428143 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:48:58.460669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.460638 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:58.461051 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.460714 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:58.474447 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.474422 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.474571 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.474478 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4pcn\" (UniqueName: \"kubernetes.io/projected/e51a0864-ff12-4b07-9a8f-678fda32f69b-kube-api-access-h4pcn\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.474636 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.474619 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.475190 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.475171 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:48:58.478958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.478941 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw"] Oct 31 00:48:58.576059 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.576033 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.576213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.576100 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4pcn\" (UniqueName: \"kubernetes.io/projected/e51a0864-ff12-4b07-9a8f-678fda32f69b-kube-api-access-h4pcn\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.576213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.576183 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.576847 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.576826 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.578476 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.578458 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.626415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.626327 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd8b12a-51d9-4205-979a-e22e81fccb68" path="/var/lib/kubelet/pods/acd8b12a-51d9-4205-979a-e22e81fccb68/volumes" Oct 31 00:48:58.634978 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.634956 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4pcn\" (UniqueName: \"kubernetes.io/projected/e51a0864-ff12-4b07-9a8f-678fda32f69b-kube-api-access-h4pcn\") pod \"maas-api-revoke-sub-5d1b8178-zhsrw\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:58.689458 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:58.689437 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:48:59.092559 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:59.092527 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw"] Oct 31 00:48:59.097626 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:48:59.097597 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51a0864_ff12_4b07_9a8f_678fda32f69b.slice/crio-a7305eaa6f021ba639e59a927d6a32727700d1fb86797ac25ae4c457099f2bb7 WatchSource:0}: Error finding container a7305eaa6f021ba639e59a927d6a32727700d1fb86797ac25ae4c457099f2bb7: Status 404 returned error can't find the container with id a7305eaa6f021ba639e59a927d6a32727700d1fb86797ac25ae4c457099f2bb7 Oct 31 00:48:59.461504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:59.461472 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:48:59.462011 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:59.461521 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 00:48:59.765818 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:59.765733 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" event={"ID":"e51a0864-ff12-4b07-9a8f-678fda32f69b","Type":"ContainerStarted","Data":"67081aa881dc2f3ae7405aa7e92c1c0b2e7045659afe0b826f2870a5e2961b95"} Oct 31 00:48:59.765818 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:59.765803 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" event={"ID":"e51a0864-ff12-4b07-9a8f-678fda32f69b","Type":"ContainerStarted","Data":"a7305eaa6f021ba639e59a927d6a32727700d1fb86797ac25ae4c457099f2bb7"} Oct 31 00:48:59.877492 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:48:59.877442 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" podStartSLOduration=1.877428603 podStartE2EDuration="1.877428603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:59.876363836 +0000 UTC m=+1221.412506142" watchObservedRunningTime="2026-10-31 00:48:59.877428603 +0000 UTC m=+1221.413570909" Oct 31 00:49:00.464937 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.464910 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:49:00.465347 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.465325 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:49:00.465992 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.465974 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:49:00.777167 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.777082 5719 generic.go:356] "Generic (PLEG): container finished" podID="e51a0864-ff12-4b07-9a8f-678fda32f69b" containerID="67081aa881dc2f3ae7405aa7e92c1c0b2e7045659afe0b826f2870a5e2961b95" exitCode=0 Oct 31 00:49:00.777333 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.777176 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" event={"ID":"e51a0864-ff12-4b07-9a8f-678fda32f69b","Type":"ContainerDied","Data":"67081aa881dc2f3ae7405aa7e92c1c0b2e7045659afe0b826f2870a5e2961b95"} Oct 31 00:49:00.808453 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.808422 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg"] Oct 31 00:49:00.813359 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.813337 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:00.869931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.869901 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg"] Oct 31 00:49:00.904505 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.904465 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wnk8\" (UniqueName: \"kubernetes.io/projected/b150081f-6e67-4690-a2f8-f3a27987376b-kube-api-access-9wnk8\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:00.904686 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.904588 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:00.904910 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:00.904870 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:01.005893 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.005865 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:01.006063 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.005947 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wnk8\" (UniqueName: \"kubernetes.io/projected/b150081f-6e67-4690-a2f8-f3a27987376b-kube-api-access-9wnk8\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:01.006063 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.005991 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:01.006589 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.006560 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:01.008234 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.008213 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:01.064894 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.064837 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wnk8\" (UniqueName: \"kubernetes.io/projected/b150081f-6e67-4690-a2f8-f3a27987376b-kube-api-access-9wnk8\") pod \"maas-api-revoke-sub-b06c3575-fr6gg\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:01.126985 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.126961 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:01.503436 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.503405 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg"] Oct 31 00:49:01.507370 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:49:01.507340 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb150081f_6e67_4690_a2f8_f3a27987376b.slice/crio-c5cacb453a510e329884379962c5425396070601df111979177d688db3b0dde7 WatchSource:0}: Error finding container c5cacb453a510e329884379962c5425396070601df111979177d688db3b0dde7: Status 404 returned error can't find the container with id c5cacb453a510e329884379962c5425396070601df111979177d688db3b0dde7 Oct 31 00:49:01.785873 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.785803 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" event={"ID":"b150081f-6e67-4690-a2f8-f3a27987376b","Type":"ContainerStarted","Data":"c5cacb453a510e329884379962c5425396070601df111979177d688db3b0dde7"} Oct 31 00:49:01.946375 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:01.946348 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:49:02.018048 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.018022 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51a0864-ff12-4b07-9a8f-678fda32f69b-kube-api-access-h4pcn\" (UniqueName: \"kubernetes.io/projected/e51a0864-ff12-4b07-9a8f-678fda32f69b-kube-api-access-h4pcn\") pod \"e51a0864-ff12-4b07-9a8f-678fda32f69b\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " Oct 31 00:49:02.018177 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.018133 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-xks-ca\") pod \"e51a0864-ff12-4b07-9a8f-678fda32f69b\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " Oct 31 00:49:02.018229 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.018195 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-openshift-ca\") pod \"e51a0864-ff12-4b07-9a8f-678fda32f69b\" (UID: \"e51a0864-ff12-4b07-9a8f-678fda32f69b\") " Oct 31 00:49:02.018785 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.018758 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-openshift-ca" pod "e51a0864-ff12-4b07-9a8f-678fda32f69b" (UID: "e51a0864-ff12-4b07-9a8f-678fda32f69b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:02.020523 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.020498 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-xks-ca" pod "e51a0864-ff12-4b07-9a8f-678fda32f69b" (UID: "e51a0864-ff12-4b07-9a8f-678fda32f69b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:02.020917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.020894 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51a0864-ff12-4b07-9a8f-678fda32f69b-kube-api-access-h4pcn" pod "e51a0864-ff12-4b07-9a8f-678fda32f69b" (UID: "e51a0864-ff12-4b07-9a8f-678fda32f69b"). InnerVolumeSpecName "kube-api-access-h4pcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:02.052211 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.052153 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw"] Oct 31 00:49:02.055323 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.055302 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66b75af9-k6pzw"] Oct 31 00:49:02.119753 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.119724 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4pcn\" (UniqueName: \"kubernetes.io/projected/e51a0864-ff12-4b07-9a8f-678fda32f69b-kube-api-access-h4pcn\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.119753 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.119748 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.119915 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.119762 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e51a0864-ff12-4b07-9a8f-678fda32f69b-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:02.609603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.609559 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cfe3cf1-23c0-491c-b6af-423989fd6a9b" path="/var/lib/kubelet/pods/5cfe3cf1-23c0-491c-b6af-423989fd6a9b/volumes" Oct 31 00:49:02.796781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.796658 5719 generic.go:356] "Generic (PLEG): container finished" podID="b150081f-6e67-4690-a2f8-f3a27987376b" containerID="7e27711ba5fb6f95c315b44a2fec5980453475f9ac47510f60c2cb15c0d7a244" exitCode=0 Oct 31 00:49:02.796781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.796754 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" event={"ID":"b150081f-6e67-4690-a2f8-f3a27987376b","Type":"ContainerDied","Data":"7e27711ba5fb6f95c315b44a2fec5980453475f9ac47510f60c2cb15c0d7a244"} Oct 31 00:49:02.798508 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.798474 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" event={"ID":"e51a0864-ff12-4b07-9a8f-678fda32f69b","Type":"ContainerDied","Data":"a7305eaa6f021ba639e59a927d6a32727700d1fb86797ac25ae4c457099f2bb7"} Oct 31 00:49:02.798633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.798523 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7305eaa6f021ba639e59a927d6a32727700d1fb86797ac25ae4c457099f2bb7" Oct 31 00:49:02.798633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:02.798486 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw" Oct 31 00:49:03.009981 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:49:03.009951 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:49:03.953347 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:03.953312 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:04.041917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.041891 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-xks-ca\") pod \"b150081f-6e67-4690-a2f8-f3a27987376b\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " Oct 31 00:49:04.042046 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.041987 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-openshift-ca\") pod \"b150081f-6e67-4690-a2f8-f3a27987376b\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " Oct 31 00:49:04.042091 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.042061 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b150081f-6e67-4690-a2f8-f3a27987376b-kube-api-access-9wnk8\" (UniqueName: \"kubernetes.io/projected/b150081f-6e67-4690-a2f8-f3a27987376b-kube-api-access-9wnk8\") pod \"b150081f-6e67-4690-a2f8-f3a27987376b\" (UID: \"b150081f-6e67-4690-a2f8-f3a27987376b\") " Oct 31 00:49:04.042406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.042369 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-openshift-ca" pod "b150081f-6e67-4690-a2f8-f3a27987376b" (UID: "b150081f-6e67-4690-a2f8-f3a27987376b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:04.042820 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.042746 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.044197 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.044172 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-xks-ca" pod "b150081f-6e67-4690-a2f8-f3a27987376b" (UID: "b150081f-6e67-4690-a2f8-f3a27987376b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:04.044292 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.044233 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b150081f-6e67-4690-a2f8-f3a27987376b-kube-api-access-9wnk8" pod "b150081f-6e67-4690-a2f8-f3a27987376b" (UID: "b150081f-6e67-4690-a2f8-f3a27987376b"). InnerVolumeSpecName "kube-api-access-9wnk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:04.144299 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.144260 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b150081f-6e67-4690-a2f8-f3a27987376b-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.144299 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.144298 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wnk8\" (UniqueName: \"kubernetes.io/projected/b150081f-6e67-4690-a2f8-f3a27987376b-kube-api-access-9wnk8\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:04.329748 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.329672 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb"] Oct 31 00:49:04.330666 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.330614 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e51a0864-ff12-4b07-9a8f-678fda32f69b" containerName="revoke-keys" Oct 31 00:49:04.330666 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.330642 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51a0864-ff12-4b07-9a8f-678fda32f69b" containerName="revoke-keys" Oct 31 00:49:04.330666 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.330668 5719 cpu_manager.go:414] "containerMap: removing container" podUID="b150081f-6e67-4690-a2f8-f3a27987376b" containerName="revoke-keys" Oct 31 00:49:04.331127 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.330676 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b150081f-6e67-4690-a2f8-f3a27987376b" containerName="revoke-keys" Oct 31 00:49:04.331127 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.330885 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51a0864-ff12-4b07-9a8f-678fda32f69b" containerName="revoke-keys" Oct 31 00:49:04.331127 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.330901 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="b150081f-6e67-4690-a2f8-f3a27987376b" containerName="revoke-keys" Oct 31 00:49:04.334951 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.334933 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.387739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.387686 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb"] Oct 31 00:49:04.447133 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.447050 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8p7q\" (UniqueName: \"kubernetes.io/projected/5a1bddbf-0480-4544-8efa-342c145cd2cc-kube-api-access-j8p7q\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.447298 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.447201 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.447359 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.447295 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.549035 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.549005 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8p7q\" (UniqueName: \"kubernetes.io/projected/5a1bddbf-0480-4544-8efa-342c145cd2cc-kube-api-access-j8p7q\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.549182 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.549093 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.549182 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.549150 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.549739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.549685 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.551742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.551719 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.607636 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.607614 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8p7q\" (UniqueName: \"kubernetes.io/projected/5a1bddbf-0480-4544-8efa-342c145cd2cc-kube-api-access-j8p7q\") pod \"maas-api-revoke-sub-1e1e00a7-jfxcb\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.648818 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.648799 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:04.817156 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.817126 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" event={"ID":"b150081f-6e67-4690-a2f8-f3a27987376b","Type":"ContainerDied","Data":"c5cacb453a510e329884379962c5425396070601df111979177d688db3b0dde7"} Oct 31 00:49:04.817297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.817175 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5cacb453a510e329884379962c5425396070601df111979177d688db3b0dde7" Oct 31 00:49:04.817297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:04.817140 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg" Oct 31 00:49:05.020931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:05.020868 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb"] Oct 31 00:49:05.025351 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:49:05.025326 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1bddbf_0480_4544_8efa_342c145cd2cc.slice/crio-b22324f7dc6e0330583dfe5201ed5ca986ce315429b297d62f912d655c2daf2c WatchSource:0}: Error finding container b22324f7dc6e0330583dfe5201ed5ca986ce315429b297d62f912d655c2daf2c: Status 404 returned error can't find the container with id b22324f7dc6e0330583dfe5201ed5ca986ce315429b297d62f912d655c2daf2c Oct 31 00:49:05.825207 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:05.825182 5719 generic.go:356] "Generic (PLEG): container finished" podID="5a1bddbf-0480-4544-8efa-342c145cd2cc" containerID="2bfb657d151ef239667e339617de1a54d4f088f9602bfd88ac34aee5c2ebff03" exitCode=0 Oct 31 00:49:05.825319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:05.825260 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" event={"ID":"5a1bddbf-0480-4544-8efa-342c145cd2cc","Type":"ContainerDied","Data":"2bfb657d151ef239667e339617de1a54d4f088f9602bfd88ac34aee5c2ebff03"} Oct 31 00:49:05.825319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:05.825305 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" event={"ID":"5a1bddbf-0480-4544-8efa-342c145cd2cc","Type":"ContainerStarted","Data":"b22324f7dc6e0330583dfe5201ed5ca986ce315429b297d62f912d655c2daf2c"} Oct 31 00:49:06.986252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:06.986229 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:07.078135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.078107 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1bddbf-0480-4544-8efa-342c145cd2cc-kube-api-access-j8p7q\" (UniqueName: \"kubernetes.io/projected/5a1bddbf-0480-4544-8efa-342c145cd2cc-kube-api-access-j8p7q\") pod \"5a1bddbf-0480-4544-8efa-342c145cd2cc\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " Oct 31 00:49:07.078264 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.078249 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-xks-ca\") pod \"5a1bddbf-0480-4544-8efa-342c145cd2cc\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " Oct 31 00:49:07.078328 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.078279 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-openshift-ca\") pod \"5a1bddbf-0480-4544-8efa-342c145cd2cc\" (UID: \"5a1bddbf-0480-4544-8efa-342c145cd2cc\") " Oct 31 00:49:07.078670 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.078639 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-openshift-ca" pod "5a1bddbf-0480-4544-8efa-342c145cd2cc" (UID: "5a1bddbf-0480-4544-8efa-342c145cd2cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:07.080287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.080261 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1bddbf-0480-4544-8efa-342c145cd2cc-kube-api-access-j8p7q" pod "5a1bddbf-0480-4544-8efa-342c145cd2cc" (UID: "5a1bddbf-0480-4544-8efa-342c145cd2cc"). InnerVolumeSpecName "kube-api-access-j8p7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:07.080408 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.080332 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-xks-ca" pod "5a1bddbf-0480-4544-8efa-342c145cd2cc" (UID: "5a1bddbf-0480-4544-8efa-342c145cd2cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:07.180507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.180436 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8p7q\" (UniqueName: \"kubernetes.io/projected/5a1bddbf-0480-4544-8efa-342c145cd2cc-kube-api-access-j8p7q\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:07.180507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.180463 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:07.180507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.180473 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1bddbf-0480-4544-8efa-342c145cd2cc-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:07.846138 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.846113 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" Oct 31 00:49:07.846138 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.846121 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb" event={"ID":"5a1bddbf-0480-4544-8efa-342c145cd2cc","Type":"ContainerDied","Data":"b22324f7dc6e0330583dfe5201ed5ca986ce315429b297d62f912d655c2daf2c"} Oct 31 00:49:07.846343 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:07.846170 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b22324f7dc6e0330583dfe5201ed5ca986ce315429b297d62f912d655c2daf2c" Oct 31 00:49:11.022509 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:11.022478 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:49:13.490717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:13.490641 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:49:13.507180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:13.506826 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:13.570827 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:13.570781 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:13.579490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:13.578780 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:13.588537 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:13.587785 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:13.610313 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:13.610275 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:13.798893 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:13.798790 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.492826 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:14.492779 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.494153 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:14.494128 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:49:14.585832 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:14.585788 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:49:14.587572 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:14.586518 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:49:14.588261 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:14.588232 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:14.589423 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:14.588739 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:49:14.618718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:14.613766 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:49:14.680719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:14.676721 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:49:15.316285 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.314778 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.334718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.316516 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.466006 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.464822 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 00:49:15.509630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.497781 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:49:15.572992 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.571991 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.615836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.614645 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:15.617025 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.616076 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 31 00:49:15.618430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.617552 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:49:15.933893 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:15.933786 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.009059 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.009003 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.170730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.169822 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.318772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.317863 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:16.318772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.317946 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.328723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.320007 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.328723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.320513 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:49:16.418723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.416811 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:16.418723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.416969 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:16.467723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.465864 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 00:49:16.480747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.480680 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:49:16.539730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.498546 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:49:16.586773 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.585651 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:49:16.586773 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.586096 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:49:16.666720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.618778 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:49:16.666720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.619231 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 31 00:49:16.666720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:16.619582 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:49:17.019052 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.018955 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.139912 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.139866 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:49:17.215090 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.214958 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.269170 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.269083 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9"] Oct 31 00:49:17.270213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.270184 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5a1bddbf-0480-4544-8efa-342c145cd2cc" containerName="revoke-keys" Oct 31 00:49:17.270371 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.270357 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1bddbf-0480-4544-8efa-342c145cd2cc" containerName="revoke-keys" Oct 31 00:49:17.270665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.270650 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1bddbf-0480-4544-8efa-342c145cd2cc" containerName="revoke-keys" Oct 31 00:49:17.298125 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.298085 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9"] Oct 31 00:49:17.298483 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.298400 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.344717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.339770 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.383726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.372784 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:49:17.383726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.372860 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:17.383726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.372930 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:17.383726 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.373023 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:17.531044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.484824 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.531044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.486192 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhbp\" (UniqueName: \"kubernetes.io/projected/b8582bf9-7d29-473f-9274-260270f96b96-kube-api-access-9lhbp\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.531044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.495836 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.531044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.496018 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.531044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.515475 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:49:17.564191 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.563858 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.594070 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.594033 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.623527 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.619440 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.623952 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.619747 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.624295 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.619772 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:49:17.624637 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.621557 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhbp\" (UniqueName: \"kubernetes.io/projected/b8582bf9-7d29-473f-9274-260270f96b96-kube-api-access-9lhbp\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.624893 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.624875 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.625748 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.625729 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.627078 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.626848 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.657229 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.657202 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kj5vp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:17.657416 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.657390 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.667713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.664144 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.801880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.801783 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:17.802412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.801877 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhbp\" (UniqueName: \"kubernetes.io/projected/b8582bf9-7d29-473f-9274-260270f96b96-kube-api-access-9lhbp\") pod \"maas-api-revoke-sub-41499fca-v94j9\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:17.802585 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.801919 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 31 00:49:17.892626 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.891294 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:49:17.992247 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:17.992215 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:18.018213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.015261 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:49:18.154106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.154066 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:49:18.197738 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.196085 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:49:18.197738 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.197095 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:49:18.216735 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.211021 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:49:18.228465 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.227602 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n"] Oct 31 00:49:18.246307 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.245247 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-872083d7-tzs6n"] Oct 31 00:49:18.247792 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.247774 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:49:18.605853 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.605817 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef697917-be6a-441d-9965-82594e1a9a58" path="/var/lib/kubelet/pods/ef697917-be6a-441d-9965-82594e1a9a58/volumes" Oct 31 00:49:18.681968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:18.681896 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9"] Oct 31 00:49:18.686119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:49:18.686092 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8582bf9_7d29_473f_9274_260270f96b96.slice/crio-38fc9c288ecedfa9daeebfc47c0c1a289662b1567e5a0371b7a3825a3f40e5b2 WatchSource:0}: Error finding container 38fc9c288ecedfa9daeebfc47c0c1a289662b1567e5a0371b7a3825a3f40e5b2: Status 404 returned error can't find the container with id 38fc9c288ecedfa9daeebfc47c0c1a289662b1567e5a0371b7a3825a3f40e5b2 Oct 31 00:49:19.373014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:19.372974 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" event={"ID":"b8582bf9-7d29-473f-9274-260270f96b96","Type":"ContainerStarted","Data":"bcbaea7b6c71c0bac71bf1edde5e78c527f42136d301c6f48fa281e2fff78939"} Oct 31 00:49:19.373185 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:19.373073 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" event={"ID":"b8582bf9-7d29-473f-9274-260270f96b96","Type":"ContainerStarted","Data":"38fc9c288ecedfa9daeebfc47c0c1a289662b1567e5a0371b7a3825a3f40e5b2"} Oct 31 00:49:19.484511 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:19.484462 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" podStartSLOduration=2.484444757 podStartE2EDuration="2.484444757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:19.483256684 +0000 UTC m=+1241.019398989" watchObservedRunningTime="2026-10-31 00:49:19.484444757 +0000 UTC m=+1241.020587061" Oct 31 00:49:20.380450 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:20.380420 5719 generic.go:356] "Generic (PLEG): container finished" podID="b8582bf9-7d29-473f-9274-260270f96b96" containerID="bcbaea7b6c71c0bac71bf1edde5e78c527f42136d301c6f48fa281e2fff78939" exitCode=0 Oct 31 00:49:20.380935 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:20.380504 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" event={"ID":"b8582bf9-7d29-473f-9274-260270f96b96","Type":"ContainerDied","Data":"bcbaea7b6c71c0bac71bf1edde5e78c527f42136d301c6f48fa281e2fff78939"} Oct 31 00:49:21.537804 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.537781 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:21.606518 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.606493 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-openshift-ca\") pod \"b8582bf9-7d29-473f-9274-260270f96b96\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " Oct 31 00:49:21.606653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.606530 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-xks-ca\") pod \"b8582bf9-7d29-473f-9274-260270f96b96\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " Oct 31 00:49:21.606653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.606583 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8582bf9-7d29-473f-9274-260270f96b96-kube-api-access-9lhbp\" (UniqueName: \"kubernetes.io/projected/b8582bf9-7d29-473f-9274-260270f96b96-kube-api-access-9lhbp\") pod \"b8582bf9-7d29-473f-9274-260270f96b96\" (UID: \"b8582bf9-7d29-473f-9274-260270f96b96\") " Oct 31 00:49:21.606901 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.606871 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-openshift-ca" pod "b8582bf9-7d29-473f-9274-260270f96b96" (UID: "b8582bf9-7d29-473f-9274-260270f96b96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:21.606969 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.606952 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:21.608756 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.608734 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-xks-ca" pod "b8582bf9-7d29-473f-9274-260270f96b96" (UID: "b8582bf9-7d29-473f-9274-260270f96b96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:21.608863 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.608805 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8582bf9-7d29-473f-9274-260270f96b96-kube-api-access-9lhbp" pod "b8582bf9-7d29-473f-9274-260270f96b96" (UID: "b8582bf9-7d29-473f-9274-260270f96b96"). InnerVolumeSpecName "kube-api-access-9lhbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:21.707983 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.707921 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8582bf9-7d29-473f-9274-260270f96b96-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:21.707983 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:21.707946 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lhbp\" (UniqueName: \"kubernetes.io/projected/b8582bf9-7d29-473f-9274-260270f96b96-kube-api-access-9lhbp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:22.394457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:22.394426 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" Oct 31 00:49:22.394457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:22.394453 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9" event={"ID":"b8582bf9-7d29-473f-9274-260270f96b96","Type":"ContainerDied","Data":"38fc9c288ecedfa9daeebfc47c0c1a289662b1567e5a0371b7a3825a3f40e5b2"} Oct 31 00:49:22.394744 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:22.394496 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38fc9c288ecedfa9daeebfc47c0c1a289662b1567e5a0371b7a3825a3f40e5b2" Oct 31 00:49:24.518001 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.517960 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm"] Oct 31 00:49:24.518827 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.518807 5719 cpu_manager.go:414] "containerMap: removing container" podUID="b8582bf9-7d29-473f-9274-260270f96b96" containerName="revoke-keys" Oct 31 00:49:24.518897 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.518830 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8582bf9-7d29-473f-9274-260270f96b96" containerName="revoke-keys" Oct 31 00:49:24.519018 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.519004 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8582bf9-7d29-473f-9274-260270f96b96" containerName="revoke-keys" Oct 31 00:49:24.522680 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.522657 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.577642 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.577608 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm"] Oct 31 00:49:24.619873 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.619844 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:24.620044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.619880 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:24.620044 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.619892 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:24.620630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.620604 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:49:24.639311 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.639277 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.639608 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.639588 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.639769 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.639754 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cjqg\" (UniqueName: \"kubernetes.io/projected/3712b2fb-38cf-4576-a36e-ef6235104a7a-kube-api-access-2cjqg\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.741081 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.741049 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.741081 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.741086 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cjqg\" (UniqueName: \"kubernetes.io/projected/3712b2fb-38cf-4576-a36e-ef6235104a7a-kube-api-access-2cjqg\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.741319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.741253 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.742055 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.742024 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.743578 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.743553 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.799826 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.799759 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cjqg\" (UniqueName: \"kubernetes.io/projected/3712b2fb-38cf-4576-a36e-ef6235104a7a-kube-api-access-2cjqg\") pod \"maas-api-revoke-sub-fa833666-9mtqm\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:24.839082 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:24.839054 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:25.225794 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:25.225764 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm"] Oct 31 00:49:25.229829 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:49:25.229800 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3712b2fb_38cf_4576_a36e_ef6235104a7a.slice/crio-bde07facde39aea8d5884dba2e1eb2c1e539713fa71a023b7c6598b1b0a2716c WatchSource:0}: Error finding container bde07facde39aea8d5884dba2e1eb2c1e539713fa71a023b7c6598b1b0a2716c: Status 404 returned error can't find the container with id bde07facde39aea8d5884dba2e1eb2c1e539713fa71a023b7c6598b1b0a2716c Oct 31 00:49:25.417026 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:25.416997 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" event={"ID":"3712b2fb-38cf-4576-a36e-ef6235104a7a","Type":"ContainerStarted","Data":"bde07facde39aea8d5884dba2e1eb2c1e539713fa71a023b7c6598b1b0a2716c"} Oct 31 00:49:26.425674 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:26.425641 5719 generic.go:356] "Generic (PLEG): container finished" podID="3712b2fb-38cf-4576-a36e-ef6235104a7a" containerID="c43c20f0720afc5539bcaf4731aebbab31314f2f75c111a868a1c8b641854672" exitCode=0 Oct 31 00:49:26.426062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:26.425731 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" event={"ID":"3712b2fb-38cf-4576-a36e-ef6235104a7a","Type":"ContainerDied","Data":"c43c20f0720afc5539bcaf4731aebbab31314f2f75c111a868a1c8b641854672"} Oct 31 00:49:27.580054 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.580031 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:27.676385 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.676353 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-xks-ca\") pod \"3712b2fb-38cf-4576-a36e-ef6235104a7a\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " Oct 31 00:49:27.676521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.676389 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-openshift-ca\") pod \"3712b2fb-38cf-4576-a36e-ef6235104a7a\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " Oct 31 00:49:27.676521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.676482 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3712b2fb-38cf-4576-a36e-ef6235104a7a-kube-api-access-2cjqg\" (UniqueName: \"kubernetes.io/projected/3712b2fb-38cf-4576-a36e-ef6235104a7a-kube-api-access-2cjqg\") pod \"3712b2fb-38cf-4576-a36e-ef6235104a7a\" (UID: \"3712b2fb-38cf-4576-a36e-ef6235104a7a\") " Oct 31 00:49:27.676822 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.676793 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-openshift-ca" pod "3712b2fb-38cf-4576-a36e-ef6235104a7a" (UID: "3712b2fb-38cf-4576-a36e-ef6235104a7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:27.676960 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.676940 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:27.678684 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.678655 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3712b2fb-38cf-4576-a36e-ef6235104a7a-kube-api-access-2cjqg" pod "3712b2fb-38cf-4576-a36e-ef6235104a7a" (UID: "3712b2fb-38cf-4576-a36e-ef6235104a7a"). InnerVolumeSpecName "kube-api-access-2cjqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:27.678684 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.678655 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-xks-ca" pod "3712b2fb-38cf-4576-a36e-ef6235104a7a" (UID: "3712b2fb-38cf-4576-a36e-ef6235104a7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:27.778461 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.778428 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cjqg\" (UniqueName: \"kubernetes.io/projected/3712b2fb-38cf-4576-a36e-ef6235104a7a-kube-api-access-2cjqg\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:27.778461 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:27.778457 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3712b2fb-38cf-4576-a36e-ef6235104a7a-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:28.048484 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:28.048399 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72"] Oct 31 00:49:28.051425 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:28.051398 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d71785-lzw72"] Oct 31 00:49:28.439372 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:28.439344 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" Oct 31 00:49:28.439372 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:28.439349 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm" event={"ID":"3712b2fb-38cf-4576-a36e-ef6235104a7a","Type":"ContainerDied","Data":"bde07facde39aea8d5884dba2e1eb2c1e539713fa71a023b7c6598b1b0a2716c"} Oct 31 00:49:28.439626 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:28.439405 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bde07facde39aea8d5884dba2e1eb2c1e539713fa71a023b7c6598b1b0a2716c" Oct 31 00:49:28.602357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:28.602326 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbbae9a-9465-478b-9cff-621287e14f77" path="/var/lib/kubelet/pods/bbbbae9a-9465-478b-9cff-621287e14f77/volumes" Oct 31 00:49:29.960554 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:29.960524 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4"] Oct 31 00:49:29.961309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:29.961261 5719 cpu_manager.go:414] "containerMap: removing container" podUID="3712b2fb-38cf-4576-a36e-ef6235104a7a" containerName="revoke-keys" Oct 31 00:49:29.961309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:29.961308 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3712b2fb-38cf-4576-a36e-ef6235104a7a" containerName="revoke-keys" Oct 31 00:49:29.961500 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:29.961485 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="3712b2fb-38cf-4576-a36e-ef6235104a7a" containerName="revoke-keys" Oct 31 00:49:29.965188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:29.965165 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.067595 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.067563 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4"] Oct 31 00:49:30.105995 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.105967 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.106160 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.106006 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.106160 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.106081 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.106160 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.106126 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.106160 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.106153 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.106296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.106258 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.106296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.106284 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.106361 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.106318 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kcm\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-kube-api-access-r8kcm\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.106392 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.106386 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208620 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208595 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208787 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208629 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208787 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208653 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208787 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208711 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208787 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208756 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208986 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208806 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208986 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208835 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208986 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208861 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kcm\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-kube-api-access-r8kcm\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.208986 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.208939 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.209180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.209030 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.209180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.209092 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.209380 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.209361 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.209540 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.209518 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.209636 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.209617 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.211162 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.211104 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.211398 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.211375 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.268686 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.268654 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.268898 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.268877 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kcm\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-kube-api-access-r8kcm\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.278683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.278664 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:30.707567 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.707528 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4"] Oct 31 00:49:30.711534 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:49:30.711505 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e9f4cf_d892_472c_9504_ed0c33ff8fff.slice/crio-82cc22dbe9c64deb18c4998cf6dda389dd8802f26d6c12cbf24805a4493a0962 WatchSource:0}: Error finding container 82cc22dbe9c64deb18c4998cf6dda389dd8802f26d6c12cbf24805a4493a0962: Status 404 returned error can't find the container with id 82cc22dbe9c64deb18c4998cf6dda389dd8802f26d6c12cbf24805a4493a0962 Oct 31 00:49:30.713843 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.713812 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:30.713905 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.713877 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:30.713948 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:30.713906 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:31.460103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:31.460068 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" event={"ID":"05e9f4cf-d892-472c-9504-ed0c33ff8fff","Type":"ContainerStarted","Data":"212350ae167789d808d67663abe422e14c7744a2649b0abcb99e45d0a7bb3d95"} Oct 31 00:49:31.460635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:31.460145 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" event={"ID":"05e9f4cf-d892-472c-9504-ed0c33ff8fff","Type":"ContainerStarted","Data":"82cc22dbe9c64deb18c4998cf6dda389dd8802f26d6c12cbf24805a4493a0962"} Oct 31 00:49:31.673387 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:31.673318 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podStartSLOduration=2.673300358 podStartE2EDuration="2.673300358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:31.646372876 +0000 UTC m=+1253.182515181" watchObservedRunningTime="2026-10-31 00:49:31.673300358 +0000 UTC m=+1253.209442664" Oct 31 00:49:32.279427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:32.279396 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:32.280965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:32.280934 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:32.281092 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:32.280983 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:33.030405 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:49:33.030369 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:49:33.279570 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.279541 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:33.279776 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.279590 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:33.626649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.626612 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x"] Oct 31 00:49:33.631655 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.631627 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.698446 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.698411 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x"] Oct 31 00:49:33.724021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.723987 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:49:33.724417 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.724394 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:49:33.724605 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.724588 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:49:33.724878 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.724862 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:49:33.770162 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.770115 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.770594 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.770573 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9bk\" (UniqueName: \"kubernetes.io/projected/f46a6850-515a-4ced-a053-75c4ec059a56-kube-api-access-nw9bk\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.771330 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.771311 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.873072 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.873043 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9bk\" (UniqueName: \"kubernetes.io/projected/f46a6850-515a-4ced-a053-75c4ec059a56-kube-api-access-nw9bk\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.873371 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.873358 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.873580 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.873566 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.874080 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.874060 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.876421 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.876394 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.932386 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.932313 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9bk\" (UniqueName: \"kubernetes.io/projected/f46a6850-515a-4ced-a053-75c4ec059a56-kube-api-access-nw9bk\") pod \"maas-api-revoke-sub-fcf4984c-pg82x\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:33.947929 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:33.947899 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:34.279139 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:34.279066 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:34.279139 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:34.279114 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:34.371767 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:34.371738 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x"] Oct 31 00:49:34.376521 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:49:34.376485 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46a6850_515a_4ced_a053_75c4ec059a56.slice/crio-ae4ef7d0e69daeaa683c6e5a40b270fcfee5b36af8a564ffcc565e9ed0707799 WatchSource:0}: Error finding container ae4ef7d0e69daeaa683c6e5a40b270fcfee5b36af8a564ffcc565e9ed0707799: Status 404 returned error can't find the container with id ae4ef7d0e69daeaa683c6e5a40b270fcfee5b36af8a564ffcc565e9ed0707799 Oct 31 00:49:34.484534 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:34.484500 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" event={"ID":"f46a6850-515a-4ced-a053-75c4ec059a56","Type":"ContainerStarted","Data":"ae4ef7d0e69daeaa683c6e5a40b270fcfee5b36af8a564ffcc565e9ed0707799"} Oct 31 00:49:35.250078 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.250050 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc"] Oct 31 00:49:35.254092 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.254075 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.279955 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.279931 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:35.280251 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.279971 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:35.330739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.329848 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc"] Oct 31 00:49:35.390263 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.390232 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.390431 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.390301 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhw2\" (UniqueName: \"kubernetes.io/projected/bc7f695e-4522-48cf-8a15-9c0e6a412d37-kube-api-access-7dhw2\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.390558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.390536 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.492102 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.492076 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.492265 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.492173 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.492265 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.492234 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhw2\" (UniqueName: \"kubernetes.io/projected/bc7f695e-4522-48cf-8a15-9c0e6a412d37-kube-api-access-7dhw2\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.493024 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.492993 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.493769 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.493739 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" event={"ID":"f46a6850-515a-4ced-a053-75c4ec059a56","Type":"ContainerStarted","Data":"2d1c79122ce759220909d2a56dea6cba16e146b1c3fddbbae3e2b59fc4a91ee4"} Oct 31 00:49:35.494542 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.494517 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.551625 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.551566 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhw2\" (UniqueName: \"kubernetes.io/projected/bc7f695e-4522-48cf-8a15-9c0e6a412d37-kube-api-access-7dhw2\") pod \"maas-api-revoke-sub-509f1050-fbkbc\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.568776 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.568752 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:35.611746 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.610219 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" podStartSLOduration=2.610199897 podStartE2EDuration="2.610199897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:35.604411491 +0000 UTC m=+1257.140553797" watchObservedRunningTime="2026-10-31 00:49:35.610199897 +0000 UTC m=+1257.146342203" Oct 31 00:49:35.972907 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:35.972873 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc"] Oct 31 00:49:35.977185 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:49:35.977158 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7f695e_4522_48cf_8a15_9c0e6a412d37.slice/crio-33850493fe9c69ae4cd85120e71022a4d93022bd05ee29c4dc6828b9c2d881a6 WatchSource:0}: Error finding container 33850493fe9c69ae4cd85120e71022a4d93022bd05ee29c4dc6828b9c2d881a6: Status 404 returned error can't find the container with id 33850493fe9c69ae4cd85120e71022a4d93022bd05ee29c4dc6828b9c2d881a6 Oct 31 00:49:36.279919 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:36.279888 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:36.280088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:36.279944 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:36.526751 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:36.526653 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" event={"ID":"bc7f695e-4522-48cf-8a15-9c0e6a412d37","Type":"ContainerStarted","Data":"d0a5a4b72dee6dbe099e2406bb911ec6feaf271c6856354a4a95028c91f63be6"} Oct 31 00:49:36.526920 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:36.526759 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" event={"ID":"bc7f695e-4522-48cf-8a15-9c0e6a412d37","Type":"ContainerStarted","Data":"33850493fe9c69ae4cd85120e71022a4d93022bd05ee29c4dc6828b9c2d881a6"} Oct 31 00:49:36.528633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:36.528610 5719 generic.go:356] "Generic (PLEG): container finished" podID="f46a6850-515a-4ced-a053-75c4ec059a56" containerID="2d1c79122ce759220909d2a56dea6cba16e146b1c3fddbbae3e2b59fc4a91ee4" exitCode=0 Oct 31 00:49:36.528802 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:36.528650 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" event={"ID":"f46a6850-515a-4ced-a053-75c4ec059a56","Type":"ContainerDied","Data":"2d1c79122ce759220909d2a56dea6cba16e146b1c3fddbbae3e2b59fc4a91ee4"} Oct 31 00:49:36.666714 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:36.666043 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" podStartSLOduration=1.666026827 podStartE2EDuration="1.666026827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:36.662007586 +0000 UTC m=+1258.198149892" watchObservedRunningTime="2026-10-31 00:49:36.666026827 +0000 UTC m=+1258.202169132" Oct 31 00:49:37.053101 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:37.053076 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c"] Oct 31 00:49:37.056371 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:37.056347 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f793c98-mpx9c"] Oct 31 00:49:37.283806 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:37.279577 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 00:49:37.283806 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:37.279649 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 00:49:37.542877 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:37.542783 5719 generic.go:356] "Generic (PLEG): container finished" podID="bc7f695e-4522-48cf-8a15-9c0e6a412d37" containerID="d0a5a4b72dee6dbe099e2406bb911ec6feaf271c6856354a4a95028c91f63be6" exitCode=0 Oct 31 00:49:37.543430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:37.543404 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" event={"ID":"bc7f695e-4522-48cf-8a15-9c0e6a412d37","Type":"ContainerDied","Data":"d0a5a4b72dee6dbe099e2406bb911ec6feaf271c6856354a4a95028c91f63be6"} Oct 31 00:49:37.914226 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:37.913890 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:38.063767 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.062558 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-xks-ca\") pod \"f46a6850-515a-4ced-a053-75c4ec059a56\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " Oct 31 00:49:38.063767 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.062717 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-openshift-ca\") pod \"f46a6850-515a-4ced-a053-75c4ec059a56\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " Oct 31 00:49:38.063767 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.062805 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46a6850-515a-4ced-a053-75c4ec059a56-kube-api-access-nw9bk\" (UniqueName: \"kubernetes.io/projected/f46a6850-515a-4ced-a053-75c4ec059a56-kube-api-access-nw9bk\") pod \"f46a6850-515a-4ced-a053-75c4ec059a56\" (UID: \"f46a6850-515a-4ced-a053-75c4ec059a56\") " Oct 31 00:49:38.066062 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.066029 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-openshift-ca" pod "f46a6850-515a-4ced-a053-75c4ec059a56" (UID: "f46a6850-515a-4ced-a053-75c4ec059a56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:38.166658 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.160241 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-xks-ca" pod "f46a6850-515a-4ced-a053-75c4ec059a56" (UID: "f46a6850-515a-4ced-a053-75c4ec059a56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:38.193719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.192134 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46a6850-515a-4ced-a053-75c4ec059a56-kube-api-access-nw9bk" pod "f46a6850-515a-4ced-a053-75c4ec059a56" (UID: "f46a6850-515a-4ced-a053-75c4ec059a56"). InnerVolumeSpecName "kube-api-access-nw9bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:38.203335 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.202527 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.203335 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.202561 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46a6850-515a-4ced-a053-75c4ec059a56-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.203335 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.202579 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9bk\" (UniqueName: \"kubernetes.io/projected/f46a6850-515a-4ced-a053-75c4ec059a56-kube-api-access-nw9bk\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:38.665723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.665074 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90de1006-9d07-4a4c-b294-3fc1baadcde6" path="/var/lib/kubelet/pods/90de1006-9d07-4a4c-b294-3fc1baadcde6/volumes" Oct 31 00:49:38.676646 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.676615 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" Oct 31 00:49:38.678899 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.678863 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x" event={"ID":"f46a6850-515a-4ced-a053-75c4ec059a56","Type":"ContainerDied","Data":"ae4ef7d0e69daeaa683c6e5a40b270fcfee5b36af8a564ffcc565e9ed0707799"} Oct 31 00:49:38.679035 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.678927 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae4ef7d0e69daeaa683c6e5a40b270fcfee5b36af8a564ffcc565e9ed0707799" Oct 31 00:49:38.891290 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:38.891261 5719 scope.go:122] "RemoveContainer" containerID="ce134ebd2dc7447cc3a2b2f7f32cc7fd264fe40c3c71fdbe49a93adb4d1846c9" Oct 31 00:49:39.018183 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.015926 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.022034 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.021656 5719 scope.go:122] "RemoveContainer" containerID="992ec92973f7c11cb1fbef08f9c3f1b1fe3ff814fc25adb48b068dafac475a0b" Oct 31 00:49:39.205454 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.205361 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.268734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.257995 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.341880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.341827 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:39.342323 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.342290 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.520010 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.518879 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.572945 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.572903 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.823771 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.799525 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:49:39.885719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.869293 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.886836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.877015 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:39.921312 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.886722 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 31 00:49:39.921848 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.888011 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded" Oct 31 00:49:39.922025 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:39.920830 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.046716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.043396 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.046716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.043862 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:49:40.226719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.195560 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.226719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.204859 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:40.226719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.208323 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:49:40.226719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.208754 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:49:40.294721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.271081 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.294721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.271494 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:49:40.358871 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.358786 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:40.359134 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.359088 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.454552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.414232 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7f695e-4522-48cf-8a15-9c0e6a412d37-kube-api-access-7dhw2\" (UniqueName: \"kubernetes.io/projected/bc7f695e-4522-48cf-8a15-9c0e6a412d37-kube-api-access-7dhw2\") pod \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " Oct 31 00:49:40.454552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.414299 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-xks-ca\") pod \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " Oct 31 00:49:40.454552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.414382 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-openshift-ca\") pod \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\" (UID: \"bc7f695e-4522-48cf-8a15-9c0e6a412d37\") " Oct 31 00:49:40.468951 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.468139 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-openshift-ca" pod "bc7f695e-4522-48cf-8a15-9c0e6a412d37" (UID: "bc7f695e-4522-48cf-8a15-9c0e6a412d37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:40.568732 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.515825 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.568732 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.529799 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.568732 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.530218 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:49:40.568732 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.559888 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7f695e-4522-48cf-8a15-9c0e6a412d37-kube-api-access-7dhw2" pod "bc7f695e-4522-48cf-8a15-9c0e6a412d37" (UID: "bc7f695e-4522-48cf-8a15-9c0e6a412d37"). InnerVolumeSpecName "kube-api-access-7dhw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:40.787060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.577714 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 00:49:40.787060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.578135 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:49:40.841918 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.828946 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhw2\" (UniqueName: \"kubernetes.io/projected/bc7f695e-4522-48cf-8a15-9c0e6a412d37-kube-api-access-7dhw2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:40.888529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.886283 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 31 00:49:40.967731 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.899163 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:49:40.967731 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.933351 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.967731 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.933671 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 31 00:49:40.967731 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.947092 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:49:40.967731 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.947173 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:49:40.999366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.987852 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:40.999366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:40.988255 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:49:41.126717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.088189 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.126717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.089365 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-xks-ca" pod "bc7f695e-4522-48cf-8a15-9c0e6a412d37" (UID: "bc7f695e-4522-48cf-8a15-9c0e6a412d37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:41.247781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.242030 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7f695e-4522-48cf-8a15-9c0e6a412d37-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:41.344951 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.248014 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.346406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.296863 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 31 00:49:41.360735 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.359036 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" event={"ID":"bc7f695e-4522-48cf-8a15-9c0e6a412d37","Type":"ContainerDied","Data":"33850493fe9c69ae4cd85120e71022a4d93022bd05ee29c4dc6828b9c2d881a6"} Oct 31 00:49:41.360735 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.359100 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33850493fe9c69ae4cd85120e71022a4d93022bd05ee29c4dc6828b9c2d881a6" Oct 31 00:49:41.360735 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.359185 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc" Oct 31 00:49:41.406857 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.406185 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:41.406857 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.406305 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.563859 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.532916 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.607718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.581263 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.900717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.900277 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.954518 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.953773 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.954518 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.953789 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:41.989802 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:41.989758 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.088837 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.088783 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.306946 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.306841 5719 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.348835 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.348779 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:49:42.349289 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.349119 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.462719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.461190 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:49:42.462719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.461249 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.534715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.533775 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.591245 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.590799 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:42.634718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.630681 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 00:49:42.853683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.852497 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:49:42.868411 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.868386 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 00:49:42.891721 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.891132 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:49:42.905477 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.905454 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 00:49:42.910080 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.910063 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:49:42.923717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.923290 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 00:49:42.949824 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:42.949777 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:49:43.151890 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:43.145339 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:49:43.326066 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:43.326035 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:43.327325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:43.327302 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:43.371963 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:43.371937 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:49:43.603716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:49:43.593075 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 00:50:03.009614 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:50:03.009581 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:50:03.088064 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.088030 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv"] Oct 31 00:50:03.088647 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.088621 5719 cpu_manager.go:414] "containerMap: removing container" podUID="bc7f695e-4522-48cf-8a15-9c0e6a412d37" containerName="revoke-keys" Oct 31 00:50:03.088647 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.088646 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7f695e-4522-48cf-8a15-9c0e6a412d37" containerName="revoke-keys" Oct 31 00:50:03.088825 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.088667 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f46a6850-515a-4ced-a053-75c4ec059a56" containerName="revoke-keys" Oct 31 00:50:03.088825 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.088673 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46a6850-515a-4ced-a053-75c4ec059a56" containerName="revoke-keys" Oct 31 00:50:03.088825 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.088823 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7f695e-4522-48cf-8a15-9c0e6a412d37" containerName="revoke-keys" Oct 31 00:50:03.088944 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.088840 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46a6850-515a-4ced-a053-75c4ec059a56" containerName="revoke-keys" Oct 31 00:50:03.092618 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.092599 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.190804 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.190780 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:50:03.190965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.190909 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:50:03.191088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.191072 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:50:03.191169 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.191091 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:50:03.194568 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.194543 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv"] Oct 31 00:50:03.201264 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.201244 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwr4p\" (UniqueName: \"kubernetes.io/projected/4e444959-9adb-427a-a31a-2856063afe69-kube-api-access-lwr4p\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.201397 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.201280 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e444959-9adb-427a-a31a-2856063afe69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.201397 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.201323 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e444959-9adb-427a-a31a-2856063afe69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.302369 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.302314 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwr4p\" (UniqueName: \"kubernetes.io/projected/4e444959-9adb-427a-a31a-2856063afe69-kube-api-access-lwr4p\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.302369 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.302351 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e444959-9adb-427a-a31a-2856063afe69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.302492 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.302390 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e444959-9adb-427a-a31a-2856063afe69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.303028 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.303009 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e444959-9adb-427a-a31a-2856063afe69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.304975 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.304953 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e444959-9adb-427a-a31a-2856063afe69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.361440 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.361421 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwr4p\" (UniqueName: \"kubernetes.io/projected/4e444959-9adb-427a-a31a-2856063afe69-kube-api-access-lwr4p\") pod \"maas-api-revoke-sub-523a6550-pz6tv\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.406429 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.406406 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:03.781167 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:03.781128 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv"] Oct 31 00:50:03.785481 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:50:03.785454 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e444959_9adb_427a_a31a_2856063afe69.slice/crio-c7fc8f0fd1a93806e4b3153118091eb6fb09af77f9b075805fc06c1896a1c14b WatchSource:0}: Error finding container c7fc8f0fd1a93806e4b3153118091eb6fb09af77f9b075805fc06c1896a1c14b: Status 404 returned error can't find the container with id c7fc8f0fd1a93806e4b3153118091eb6fb09af77f9b075805fc06c1896a1c14b Oct 31 00:50:04.630553 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:04.630524 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" event={"ID":"4e444959-9adb-427a-a31a-2856063afe69","Type":"ContainerStarted","Data":"bdc1c22f7f55ac2e2fcc720b0614e339a57187d171fa56f21d8d96a1bf06b098"} Oct 31 00:50:04.630959 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:04.630572 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" event={"ID":"4e444959-9adb-427a-a31a-2856063afe69","Type":"ContainerStarted","Data":"c7fc8f0fd1a93806e4b3153118091eb6fb09af77f9b075805fc06c1896a1c14b"} Oct 31 00:50:04.744017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:04.743958 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" podStartSLOduration=1.743944152 podStartE2EDuration="1.743944152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:04.74207363 +0000 UTC m=+1286.278215938" watchObservedRunningTime="2026-10-31 00:50:04.743944152 +0000 UTC m=+1286.280086457" Oct 31 00:50:05.638625 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:05.638591 5719 generic.go:356] "Generic (PLEG): container finished" podID="4e444959-9adb-427a-a31a-2856063afe69" containerID="bdc1c22f7f55ac2e2fcc720b0614e339a57187d171fa56f21d8d96a1bf06b098" exitCode=0 Oct 31 00:50:05.639034 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:05.638677 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" event={"ID":"4e444959-9adb-427a-a31a-2856063afe69","Type":"ContainerDied","Data":"bdc1c22f7f55ac2e2fcc720b0614e339a57187d171fa56f21d8d96a1bf06b098"} Oct 31 00:50:06.797354 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:06.797325 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:06.944424 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:06.944350 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e444959-9adb-427a-a31a-2856063afe69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e444959-9adb-427a-a31a-2856063afe69-maas-api-openshift-ca\") pod \"4e444959-9adb-427a-a31a-2856063afe69\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " Oct 31 00:50:06.944424 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:06.944398 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e444959-9adb-427a-a31a-2856063afe69-kube-api-access-lwr4p\" (UniqueName: \"kubernetes.io/projected/4e444959-9adb-427a-a31a-2856063afe69-kube-api-access-lwr4p\") pod \"4e444959-9adb-427a-a31a-2856063afe69\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " Oct 31 00:50:06.944647 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:06.944456 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e444959-9adb-427a-a31a-2856063afe69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e444959-9adb-427a-a31a-2856063afe69-maas-api-xks-ca\") pod \"4e444959-9adb-427a-a31a-2856063afe69\" (UID: \"4e444959-9adb-427a-a31a-2856063afe69\") " Oct 31 00:50:06.944758 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:06.944729 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e444959-9adb-427a-a31a-2856063afe69-maas-api-openshift-ca" pod "4e444959-9adb-427a-a31a-2856063afe69" (UID: "4e444959-9adb-427a-a31a-2856063afe69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:06.944886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:06.944871 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e444959-9adb-427a-a31a-2856063afe69-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.946740 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:06.946710 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e444959-9adb-427a-a31a-2856063afe69-maas-api-xks-ca" pod "4e444959-9adb-427a-a31a-2856063afe69" (UID: "4e444959-9adb-427a-a31a-2856063afe69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:06.946873 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:06.946851 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e444959-9adb-427a-a31a-2856063afe69-kube-api-access-lwr4p" pod "4e444959-9adb-427a-a31a-2856063afe69" (UID: "4e444959-9adb-427a-a31a-2856063afe69"). InnerVolumeSpecName "kube-api-access-lwr4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:07.046283 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:07.046255 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwr4p\" (UniqueName: \"kubernetes.io/projected/4e444959-9adb-427a-a31a-2856063afe69-kube-api-access-lwr4p\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:07.046283 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:07.046280 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e444959-9adb-427a-a31a-2856063afe69-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:07.653421 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:07.653383 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" Oct 31 00:50:07.653587 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:07.653425 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv" event={"ID":"4e444959-9adb-427a-a31a-2856063afe69","Type":"ContainerDied","Data":"c7fc8f0fd1a93806e4b3153118091eb6fb09af77f9b075805fc06c1896a1c14b"} Oct 31 00:50:07.653587 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:07.653462 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7fc8f0fd1a93806e4b3153118091eb6fb09af77f9b075805fc06c1896a1c14b" Oct 31 00:50:10.420137 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:10.420094 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:50:12.950830 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:12.950781 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:12.958855 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:12.958818 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.097720 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.093764 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 00:50:13.352286 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.352240 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.353310 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.353281 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 31 00:50:13.558643 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.557850 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.584931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.583775 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.951824 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.951780 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.954222 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.954199 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:50:13.962809 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.962778 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.963943 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.962967 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:50:13.964778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:13.964489 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:50:14.466787 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:14.400430 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 00:50:14.471730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:14.400511 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 31 00:50:14.475573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:14.475251 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:50:14.475967 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:14.475950 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:50:14.489866 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:14.489805 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 00:50:14.586718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:14.585804 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 00:50:14.591716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:14.588157 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 00:50:14.594713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:14.592896 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 00:50:15.116718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.099720 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx"] Oct 31 00:50:15.116718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.108585 5719 cpu_manager.go:414] "containerMap: removing container" podUID="4e444959-9adb-427a-a31a-2856063afe69" containerName="revoke-keys" Oct 31 00:50:15.116718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.108608 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e444959-9adb-427a-a31a-2856063afe69" containerName="revoke-keys" Oct 31 00:50:15.116718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.108824 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e444959-9adb-427a-a31a-2856063afe69" containerName="revoke-keys" Oct 31 00:50:15.116718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.114200 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.186857 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.186914 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.187003 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6gfpq\"" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.187070 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.188982 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snk6d\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-kube-api-access-snk6d\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.189052 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.189132 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.189177 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.189263 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.189307 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33103902-63e1-4785-8af4-59501cb92cf4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.189329 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.192717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.190288 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx"] Oct 31 00:50:15.292371 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.292177 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.292371 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.292250 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.299425 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.292343 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.299599 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.299498 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33103902-63e1-4785-8af4-59501cb92cf4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.299868 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.299524 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.300168 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.300145 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snk6d\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-kube-api-access-snk6d\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.300371 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.300350 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.305772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.302643 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.322646 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.322360 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.345839 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.344680 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33103902-63e1-4785-8af4-59501cb92cf4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.366393 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.366336 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snk6d\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-kube-api-access-snk6d\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.376765 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.376169 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.376765 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.376186 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.378724 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.378657 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5ctsx\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.524873 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.524836 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:15.990529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.990502 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx"] Oct 31 00:50:15.992747 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:50:15.992673 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33103902_63e1_4785_8af4_59501cb92cf4.slice/crio-9d1f2162a784b1feafcc17a5eaf0d94445d7547714603058c1f5b66936492dc7 WatchSource:0}: Error finding container 9d1f2162a784b1feafcc17a5eaf0d94445d7547714603058c1f5b66936492dc7: Status 404 returned error can't find the container with id 9d1f2162a784b1feafcc17a5eaf0d94445d7547714603058c1f5b66936492dc7 Oct 31 00:50:15.995087 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:15.995066 5719 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:50:16.820558 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:16.820515 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" event={"ID":"33103902-63e1-4785-8af4-59501cb92cf4","Type":"ContainerStarted","Data":"9d1f2162a784b1feafcc17a5eaf0d94445d7547714603058c1f5b66936492dc7"} Oct 31 00:50:23.506969 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:23.506937 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:23.507267 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:23.507000 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:23.876879 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:23.876840 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" event={"ID":"33103902-63e1-4785-8af4-59501cb92cf4","Type":"ContainerStarted","Data":"2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6"} Oct 31 00:50:23.877076 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:23.877058 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:24.090031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:24.089980 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" podStartSLOduration=2.579539294 podStartE2EDuration="10.089967298s" totalImagesPullingTime="7.510428004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:24.040884083 +0000 UTC m=+1305.577026388" watchObservedRunningTime="2026-10-31 00:50:24.089967298 +0000 UTC m=+1305.626109603" Oct 31 00:50:24.895914 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:24.895862 5719 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-5ctsx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:50:24.896430 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:24.895931 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" podUID="33103902-63e1-4785-8af4-59501cb92cf4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:27.894275 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:27.894242 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 00:50:33.009398 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:50:33.009357 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:50:40.136060 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:40.136029 5719 scope.go:122] "RemoveContainer" containerID="38fe59bc24acd08642f52fb1fc23f59ef3a8ee3a427be833d7734cc4c2be3ab1" Oct 31 00:50:40.145919 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:40.145901 5719 scope.go:122] "RemoveContainer" containerID="26762eb363061b4d7ee97e19c4daa86286f7d612c5c4ab2311c78d9574e2f5fd" Oct 31 00:50:40.154853 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:40.154834 5719 scope.go:122] "RemoveContainer" containerID="747f36227a6750cda2f614795d3e5f7717080dd22244688c172446aa6d611034" Oct 31 00:50:40.163132 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:40.163112 5719 scope.go:122] "RemoveContainer" containerID="a5a4ccbf2dd017bcc2ba05192e335240d59fb15b229b2b36f1345b306db985f5" Oct 31 00:50:40.171637 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:40.171596 5719 scope.go:122] "RemoveContainer" containerID="ed5b159f048a3ab5b9b45325f5f4e2918da2ffd7f0543da44bfcf605eeabb653" Oct 31 00:50:40.180290 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:50:40.180264 5719 scope.go:122] "RemoveContainer" containerID="3274341d28d240504d6f8d6d198c81fd4ba2579647a67972f3570c1cc3a0a144" Oct 31 00:51:00.243580 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:00.243547 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:51:03.010820 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:51:03.010785 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:51:15.238723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.238667 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp"] Oct 31 00:51:15.245925 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.245899 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.298392 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.298353 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp"] Oct 31 00:51:15.330914 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.330891 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54kb\" (UniqueName: \"kubernetes.io/projected/6496ae91-5a84-451c-8e8b-9f21adf6f302-kube-api-access-g54kb\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.331052 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.331028 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.331234 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.331213 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.341169 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.341148 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:15.341340 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.341326 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:15.341340 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.341335 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:15.341443 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.341367 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:51:15.432772 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.432748 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.432881 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.432841 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.432924 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.432888 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54kb\" (UniqueName: \"kubernetes.io/projected/6496ae91-5a84-451c-8e8b-9f21adf6f302-kube-api-access-g54kb\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.433468 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.433415 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.435171 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.435154 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.492089 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.492024 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54kb\" (UniqueName: \"kubernetes.io/projected/6496ae91-5a84-451c-8e8b-9f21adf6f302-kube-api-access-g54kb\") pod \"maas-api-revoke-sub-e29508e6-p8zcp\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.559829 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.559806 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:15.936982 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:15.936951 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp"] Oct 31 00:51:15.941271 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:51:15.941249 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6496ae91_5a84_451c_8e8b_9f21adf6f302.slice/crio-a1ac56a270d3766e7be9600d6e846933d40dcf05fbe6b14e087335561caa4e76 WatchSource:0}: Error finding container a1ac56a270d3766e7be9600d6e846933d40dcf05fbe6b14e087335561caa4e76: Status 404 returned error can't find the container with id a1ac56a270d3766e7be9600d6e846933d40dcf05fbe6b14e087335561caa4e76 Oct 31 00:51:16.226541 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:16.226513 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" event={"ID":"6496ae91-5a84-451c-8e8b-9f21adf6f302","Type":"ContainerStarted","Data":"3cb65d1bd743bdc0d702e072a9f4ac773816cec4c83cb3f092c1753a9fe1590f"} Oct 31 00:51:16.226665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:16.226575 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" event={"ID":"6496ae91-5a84-451c-8e8b-9f21adf6f302","Type":"ContainerStarted","Data":"a1ac56a270d3766e7be9600d6e846933d40dcf05fbe6b14e087335561caa4e76"} Oct 31 00:51:16.340316 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:16.340270 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" podStartSLOduration=1.340258081 podStartE2EDuration="1.340258081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:16.33777832 +0000 UTC m=+1357.873920626" watchObservedRunningTime="2026-10-31 00:51:16.340258081 +0000 UTC m=+1357.876400387" Oct 31 00:51:17.234415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:17.234386 5719 generic.go:356] "Generic (PLEG): container finished" podID="6496ae91-5a84-451c-8e8b-9f21adf6f302" containerID="3cb65d1bd743bdc0d702e072a9f4ac773816cec4c83cb3f092c1753a9fe1590f" exitCode=0 Oct 31 00:51:17.234588 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:17.234425 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" event={"ID":"6496ae91-5a84-451c-8e8b-9f21adf6f302","Type":"ContainerDied","Data":"3cb65d1bd743bdc0d702e072a9f4ac773816cec4c83cb3f092c1753a9fe1590f"} Oct 31 00:51:18.392585 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.392557 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:18.468005 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.467976 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-xks-ca\") pod \"6496ae91-5a84-451c-8e8b-9f21adf6f302\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " Oct 31 00:51:18.468140 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.468022 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-openshift-ca\") pod \"6496ae91-5a84-451c-8e8b-9f21adf6f302\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " Oct 31 00:51:18.468203 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.468182 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6496ae91-5a84-451c-8e8b-9f21adf6f302-kube-api-access-g54kb\" (UniqueName: \"kubernetes.io/projected/6496ae91-5a84-451c-8e8b-9f21adf6f302-kube-api-access-g54kb\") pod \"6496ae91-5a84-451c-8e8b-9f21adf6f302\" (UID: \"6496ae91-5a84-451c-8e8b-9f21adf6f302\") " Oct 31 00:51:18.468460 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.468433 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-openshift-ca" pod "6496ae91-5a84-451c-8e8b-9f21adf6f302" (UID: "6496ae91-5a84-451c-8e8b-9f21adf6f302"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:18.468595 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.468575 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:18.470064 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.470044 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-xks-ca" pod "6496ae91-5a84-451c-8e8b-9f21adf6f302" (UID: "6496ae91-5a84-451c-8e8b-9f21adf6f302"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:18.470196 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.470176 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6496ae91-5a84-451c-8e8b-9f21adf6f302-kube-api-access-g54kb" pod "6496ae91-5a84-451c-8e8b-9f21adf6f302" (UID: "6496ae91-5a84-451c-8e8b-9f21adf6f302"). InnerVolumeSpecName "kube-api-access-g54kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:18.569305 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.569282 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g54kb\" (UniqueName: \"kubernetes.io/projected/6496ae91-5a84-451c-8e8b-9f21adf6f302-kube-api-access-g54kb\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:18.569305 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:18.569306 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6496ae91-5a84-451c-8e8b-9f21adf6f302-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:19.249882 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:19.249844 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" Oct 31 00:51:19.250049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:19.249880 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp" event={"ID":"6496ae91-5a84-451c-8e8b-9f21adf6f302","Type":"ContainerDied","Data":"a1ac56a270d3766e7be9600d6e846933d40dcf05fbe6b14e087335561caa4e76"} Oct 31 00:51:19.250049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:19.249931 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1ac56a270d3766e7be9600d6e846933d40dcf05fbe6b14e087335561caa4e76" Oct 31 00:51:31.965140 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:31.965108 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr"] Oct 31 00:51:31.965627 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:31.965613 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6496ae91-5a84-451c-8e8b-9f21adf6f302" containerName="revoke-keys" Oct 31 00:51:31.965627 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:31.965626 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6496ae91-5a84-451c-8e8b-9f21adf6f302" containerName="revoke-keys" Oct 31 00:51:31.965806 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:31.965753 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6496ae91-5a84-451c-8e8b-9f21adf6f302" containerName="revoke-keys" Oct 31 00:51:31.969337 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:31.969318 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.024619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.024594 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:32.050803 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.050778 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.050965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.050941 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjrj\" (UniqueName: \"kubernetes.io/projected/61898d97-49da-49bb-a4fc-f8bcf6cf2997-kube-api-access-dpjrj\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.051029 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.051000 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.072482 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.072462 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:32.072594 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.072487 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:32.072645 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.072630 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:51:32.076389 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.076371 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr"] Oct 31 00:51:32.152320 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.152300 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.152419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.152376 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjrj\" (UniqueName: \"kubernetes.io/projected/61898d97-49da-49bb-a4fc-f8bcf6cf2997-kube-api-access-dpjrj\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.152419 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.152404 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.152937 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.152913 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.154977 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.154956 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.211421 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.211394 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjrj\" (UniqueName: \"kubernetes.io/projected/61898d97-49da-49bb-a4fc-f8bcf6cf2997-kube-api-access-dpjrj\") pod \"maas-api-revoke-sub-0e3f42f1-vpbbr\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.283103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.283052 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:32.665146 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:32.665115 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr"] Oct 31 00:51:32.667842 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:51:32.667817 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61898d97_49da_49bb_a4fc_f8bcf6cf2997.slice/crio-6a9f39a5f32d52e7e724ddc8b67cd92b9e80f7ca40b2c80a1aa7435b05a19de2 WatchSource:0}: Error finding container 6a9f39a5f32d52e7e724ddc8b67cd92b9e80f7ca40b2c80a1aa7435b05a19de2: Status 404 returned error can't find the container with id 6a9f39a5f32d52e7e724ddc8b67cd92b9e80f7ca40b2c80a1aa7435b05a19de2 Oct 31 00:51:33.013322 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:51:33.013293 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:51:33.344173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:33.344143 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" event={"ID":"61898d97-49da-49bb-a4fc-f8bcf6cf2997","Type":"ContainerStarted","Data":"d3cd33e1f0710fce84e7cce5dae8c9078219993f24c678688aa0f24c4c01f5b1"} Oct 31 00:51:33.344357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:33.344229 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" event={"ID":"61898d97-49da-49bb-a4fc-f8bcf6cf2997","Type":"ContainerStarted","Data":"6a9f39a5f32d52e7e724ddc8b67cd92b9e80f7ca40b2c80a1aa7435b05a19de2"} Oct 31 00:51:33.457701 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:33.457654 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" podStartSLOduration=2.45764018 podStartE2EDuration="2.45764018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:33.455681407 +0000 UTC m=+1374.991823711" watchObservedRunningTime="2026-10-31 00:51:33.45764018 +0000 UTC m=+1374.993782529" Oct 31 00:51:34.351645 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:34.351611 5719 generic.go:356] "Generic (PLEG): container finished" podID="61898d97-49da-49bb-a4fc-f8bcf6cf2997" containerID="d3cd33e1f0710fce84e7cce5dae8c9078219993f24c678688aa0f24c4c01f5b1" exitCode=0 Oct 31 00:51:34.352033 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:34.351669 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" event={"ID":"61898d97-49da-49bb-a4fc-f8bcf6cf2997","Type":"ContainerDied","Data":"d3cd33e1f0710fce84e7cce5dae8c9078219993f24c678688aa0f24c4c01f5b1"} Oct 31 00:51:35.513402 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.513378 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:35.599635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.599609 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-xks-ca\") pod \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " Oct 31 00:51:35.599814 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.599684 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-openshift-ca\") pod \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " Oct 31 00:51:35.599894 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.599815 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61898d97-49da-49bb-a4fc-f8bcf6cf2997-kube-api-access-dpjrj\" (UniqueName: \"kubernetes.io/projected/61898d97-49da-49bb-a4fc-f8bcf6cf2997-kube-api-access-dpjrj\") pod \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\" (UID: \"61898d97-49da-49bb-a4fc-f8bcf6cf2997\") " Oct 31 00:51:35.600035 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.600010 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-openshift-ca" pod "61898d97-49da-49bb-a4fc-f8bcf6cf2997" (UID: "61898d97-49da-49bb-a4fc-f8bcf6cf2997"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:35.600252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.600232 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.601969 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.601945 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-xks-ca" pod "61898d97-49da-49bb-a4fc-f8bcf6cf2997" (UID: "61898d97-49da-49bb-a4fc-f8bcf6cf2997"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:35.602065 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.601977 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61898d97-49da-49bb-a4fc-f8bcf6cf2997-kube-api-access-dpjrj" pod "61898d97-49da-49bb-a4fc-f8bcf6cf2997" (UID: "61898d97-49da-49bb-a4fc-f8bcf6cf2997"). InnerVolumeSpecName "kube-api-access-dpjrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:35.701552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.701472 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpjrj\" (UniqueName: \"kubernetes.io/projected/61898d97-49da-49bb-a4fc-f8bcf6cf2997-kube-api-access-dpjrj\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:35.701552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:35.701502 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61898d97-49da-49bb-a4fc-f8bcf6cf2997-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:36.365446 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:36.365406 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" event={"ID":"61898d97-49da-49bb-a4fc-f8bcf6cf2997","Type":"ContainerDied","Data":"6a9f39a5f32d52e7e724ddc8b67cd92b9e80f7ca40b2c80a1aa7435b05a19de2"} Oct 31 00:51:36.365446 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:36.365444 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr" Oct 31 00:51:36.365651 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:36.365496 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a9f39a5f32d52e7e724ddc8b67cd92b9e80f7ca40b2c80a1aa7435b05a19de2" Oct 31 00:51:38.110098 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.110068 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm"] Oct 31 00:51:38.110604 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.110590 5719 cpu_manager.go:414] "containerMap: removing container" podUID="61898d97-49da-49bb-a4fc-f8bcf6cf2997" containerName="revoke-keys" Oct 31 00:51:38.110649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.110604 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61898d97-49da-49bb-a4fc-f8bcf6cf2997" containerName="revoke-keys" Oct 31 00:51:38.110739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.110729 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="61898d97-49da-49bb-a4fc-f8bcf6cf2997" containerName="revoke-keys" Oct 31 00:51:38.114361 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.114346 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.168309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.168283 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:51:38.216188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.216160 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:51:38.216364 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.216173 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:51:38.216364 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.216281 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:51:38.220129 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.220099 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm"] Oct 31 00:51:38.230366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.230340 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.230477 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.230373 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.230606 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.230519 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cc8f\" (UniqueName: \"kubernetes.io/projected/01fa54b6-9b16-4614-9d5a-08955c8821b6-kube-api-access-4cc8f\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.331975 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.331948 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.332084 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.331981 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.332084 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.332051 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cc8f\" (UniqueName: \"kubernetes.io/projected/01fa54b6-9b16-4614-9d5a-08955c8821b6-kube-api-access-4cc8f\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.332610 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.332591 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.334483 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.334466 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.390982 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.390923 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cc8f\" (UniqueName: \"kubernetes.io/projected/01fa54b6-9b16-4614-9d5a-08955c8821b6-kube-api-access-4cc8f\") pod \"maas-api-revoke-sub-a994e802-54jkm\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.428100 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.428082 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:38.802868 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:38.802794 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm"] Oct 31 00:51:38.808265 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:51:38.808236 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fa54b6_9b16_4614_9d5a_08955c8821b6.slice/crio-1de8aaf4bf96939d7b4d498c8049b081ef90dc7dca2adf749de6e20ef04b52dc WatchSource:0}: Error finding container 1de8aaf4bf96939d7b4d498c8049b081ef90dc7dca2adf749de6e20ef04b52dc: Status 404 returned error can't find the container with id 1de8aaf4bf96939d7b4d498c8049b081ef90dc7dca2adf749de6e20ef04b52dc Oct 31 00:51:39.386195 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:39.386105 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" event={"ID":"01fa54b6-9b16-4614-9d5a-08955c8821b6","Type":"ContainerStarted","Data":"e6b11912d29d4cfa82f2b21bbe0b8b88044b6c7f5324448ae23d2b7e20023cde"} Oct 31 00:51:39.386589 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:39.386197 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" event={"ID":"01fa54b6-9b16-4614-9d5a-08955c8821b6","Type":"ContainerStarted","Data":"1de8aaf4bf96939d7b4d498c8049b081ef90dc7dca2adf749de6e20ef04b52dc"} Oct 31 00:51:39.498629 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:39.498578 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" podStartSLOduration=1.498565728 podStartE2EDuration="1.498565728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:39.498004905 +0000 UTC m=+1381.034147210" watchObservedRunningTime="2026-10-31 00:51:39.498565728 +0000 UTC m=+1381.034708026" Oct 31 00:51:40.394087 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:40.394056 5719 generic.go:356] "Generic (PLEG): container finished" podID="01fa54b6-9b16-4614-9d5a-08955c8821b6" containerID="e6b11912d29d4cfa82f2b21bbe0b8b88044b6c7f5324448ae23d2b7e20023cde" exitCode=0 Oct 31 00:51:40.394499 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:40.394131 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" event={"ID":"01fa54b6-9b16-4614-9d5a-08955c8821b6","Type":"ContainerDied","Data":"e6b11912d29d4cfa82f2b21bbe0b8b88044b6c7f5324448ae23d2b7e20023cde"} Oct 31 00:51:41.554630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.554607 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:41.670776 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.670680 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fa54b6-9b16-4614-9d5a-08955c8821b6-kube-api-access-4cc8f\" (UniqueName: \"kubernetes.io/projected/01fa54b6-9b16-4614-9d5a-08955c8821b6-kube-api-access-4cc8f\") pod \"01fa54b6-9b16-4614-9d5a-08955c8821b6\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " Oct 31 00:51:41.670942 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.670923 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-openshift-ca\") pod \"01fa54b6-9b16-4614-9d5a-08955c8821b6\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " Oct 31 00:51:41.671006 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.670961 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-xks-ca\") pod \"01fa54b6-9b16-4614-9d5a-08955c8821b6\" (UID: \"01fa54b6-9b16-4614-9d5a-08955c8821b6\") " Oct 31 00:51:41.671277 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.671254 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-openshift-ca" pod "01fa54b6-9b16-4614-9d5a-08955c8821b6" (UID: "01fa54b6-9b16-4614-9d5a-08955c8821b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:51:41.671611 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.671587 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:41.673120 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.673095 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fa54b6-9b16-4614-9d5a-08955c8821b6-kube-api-access-4cc8f" pod "01fa54b6-9b16-4614-9d5a-08955c8821b6" (UID: "01fa54b6-9b16-4614-9d5a-08955c8821b6"). InnerVolumeSpecName "kube-api-access-4cc8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:41.673354 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.673334 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-xks-ca" pod "01fa54b6-9b16-4614-9d5a-08955c8821b6" (UID: "01fa54b6-9b16-4614-9d5a-08955c8821b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:41.773305 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.773263 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cc8f\" (UniqueName: \"kubernetes.io/projected/01fa54b6-9b16-4614-9d5a-08955c8821b6-kube-api-access-4cc8f\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:41.773305 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:41.773298 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01fa54b6-9b16-4614-9d5a-08955c8821b6-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:42.408972 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:42.408938 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" event={"ID":"01fa54b6-9b16-4614-9d5a-08955c8821b6","Type":"ContainerDied","Data":"1de8aaf4bf96939d7b4d498c8049b081ef90dc7dca2adf749de6e20ef04b52dc"} Oct 31 00:51:42.409153 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:42.408995 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1de8aaf4bf96939d7b4d498c8049b081ef90dc7dca2adf749de6e20ef04b52dc" Oct 31 00:51:42.409153 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:42.408998 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm" Oct 31 00:51:44.103606 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:51:44.103577 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:52:03.018901 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:52:03.018855 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:52:13.841218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.841188 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj"] Oct 31 00:52:13.841763 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.841749 5719 cpu_manager.go:414] "containerMap: removing container" podUID="01fa54b6-9b16-4614-9d5a-08955c8821b6" containerName="revoke-keys" Oct 31 00:52:13.841815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.841764 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fa54b6-9b16-4614-9d5a-08955c8821b6" containerName="revoke-keys" Oct 31 00:52:13.841880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.841871 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fa54b6-9b16-4614-9d5a-08955c8821b6" containerName="revoke-keys" Oct 31 00:52:13.845656 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.845634 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:13.877940 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.877910 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdz7l\" (UniqueName: \"kubernetes.io/projected/d8984cf1-62df-4938-a90a-2a129e9b5e2d-kube-api-access-cdz7l\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:13.878117 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.878099 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:13.878188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.878138 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:13.901979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.901956 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj"] Oct 31 00:52:13.944828 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.944796 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:13.944936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.944796 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:13.944936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.944800 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:13.944936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.944851 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:52:13.979446 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.979421 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdz7l\" (UniqueName: \"kubernetes.io/projected/d8984cf1-62df-4938-a90a-2a129e9b5e2d-kube-api-access-cdz7l\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:13.979600 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.979580 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:13.979648 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.979622 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:13.980210 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.980184 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:13.982050 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:13.982034 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:14.038574 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:14.038546 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdz7l\" (UniqueName: \"kubernetes.io/projected/d8984cf1-62df-4938-a90a-2a129e9b5e2d-kube-api-access-cdz7l\") pod \"maas-api-revoke-sub-f1bb633d-qqqmj\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:14.158527 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:14.158470 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:14.533634 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:14.533556 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj"] Oct 31 00:52:14.537900 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:52:14.537866 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8984cf1_62df_4938_a90a_2a129e9b5e2d.slice/crio-11ade1a1ba6dcd22504e0bfca158338aa54a250ded5e0d8b13a66ba2d2bf08af WatchSource:0}: Error finding container 11ade1a1ba6dcd22504e0bfca158338aa54a250ded5e0d8b13a66ba2d2bf08af: Status 404 returned error can't find the container with id 11ade1a1ba6dcd22504e0bfca158338aa54a250ded5e0d8b13a66ba2d2bf08af Oct 31 00:52:14.757831 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:14.757802 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" event={"ID":"d8984cf1-62df-4938-a90a-2a129e9b5e2d","Type":"ContainerStarted","Data":"11ade1a1ba6dcd22504e0bfca158338aa54a250ded5e0d8b13a66ba2d2bf08af"} Oct 31 00:52:15.072194 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.072164 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk"] Oct 31 00:52:15.076427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.076410 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.126659 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.126640 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b63c24-openshift-default-dockercfg-58m59\"" Oct 31 00:52:15.167288 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.167264 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk"] Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193212 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193337 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193564 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193711 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193741 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193762 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bsh8\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-kube-api-access-4bsh8\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193817 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193844 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.194073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.193872 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.294553 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294522 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.294744 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294639 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.294744 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294662 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.294744 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294678 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bsh8\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-kube-api-access-4bsh8\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294767 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294795 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294824 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294862 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.294912 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295286 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.295027 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295286 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.295119 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295389 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.295336 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295389 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.295365 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.295755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.295733 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.297961 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.297935 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.298087 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.298072 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.353422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.353363 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bsh8\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-kube-api-access-4bsh8\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.353518 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.353445 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.390600 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.390576 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:15.766406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.766340 5719 generic.go:356] "Generic (PLEG): container finished" podID="d8984cf1-62df-4938-a90a-2a129e9b5e2d" containerID="e226c781ea7fe30adf33f79b6ccd30e90ef093f43f02295fd4a931c9fd1f209d" exitCode=0 Oct 31 00:52:15.766532 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.766432 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" event={"ID":"d8984cf1-62df-4938-a90a-2a129e9b5e2d","Type":"ContainerDied","Data":"e226c781ea7fe30adf33f79b6ccd30e90ef093f43f02295fd4a931c9fd1f209d"} Oct 31 00:52:15.818993 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.818970 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk"] Oct 31 00:52:15.822244 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:52:15.822223 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5ebf5e_00cf_4b8c_88d0_639967671f62.slice/crio-09545a350480198abcc2fe2e8db24783735d9145276954387b59234d78517136 WatchSource:0}: Error finding container 09545a350480198abcc2fe2e8db24783735d9145276954387b59234d78517136: Status 404 returned error can't find the container with id 09545a350480198abcc2fe2e8db24783735d9145276954387b59234d78517136 Oct 31 00:52:15.824450 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.824423 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:52:15.824524 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.824482 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:52:15.824524 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:15.824509 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:52:16.776297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:16.776267 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" event={"ID":"7f5ebf5e-00cf-4b8c-88d0-639967671f62","Type":"ContainerStarted","Data":"823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53"} Oct 31 00:52:16.776762 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:16.776318 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" event={"ID":"7f5ebf5e-00cf-4b8c-88d0-639967671f62","Type":"ContainerStarted","Data":"09545a350480198abcc2fe2e8db24783735d9145276954387b59234d78517136"} Oct 31 00:52:16.996370 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:16.996348 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:16.998442 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:16.998405 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" podStartSLOduration=1.998392409 podStartE2EDuration="1.998392409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:16.948312565 +0000 UTC m=+1418.484454871" watchObservedRunningTime="2026-10-31 00:52:16.998392409 +0000 UTC m=+1418.534534713" Oct 31 00:52:17.138066 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.138035 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8984cf1-62df-4938-a90a-2a129e9b5e2d-kube-api-access-cdz7l\" (UniqueName: \"kubernetes.io/projected/d8984cf1-62df-4938-a90a-2a129e9b5e2d-kube-api-access-cdz7l\") pod \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " Oct 31 00:52:17.138250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.138129 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-xks-ca\") pod \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " Oct 31 00:52:17.138250 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.138165 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-openshift-ca\") pod \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\" (UID: \"d8984cf1-62df-4938-a90a-2a129e9b5e2d\") " Oct 31 00:52:17.138619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.138575 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-openshift-ca" pod "d8984cf1-62df-4938-a90a-2a129e9b5e2d" (UID: "d8984cf1-62df-4938-a90a-2a129e9b5e2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:17.139263 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.139241 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:17.140524 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.140501 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8984cf1-62df-4938-a90a-2a129e9b5e2d-kube-api-access-cdz7l" pod "d8984cf1-62df-4938-a90a-2a129e9b5e2d" (UID: "d8984cf1-62df-4938-a90a-2a129e9b5e2d"). InnerVolumeSpecName "kube-api-access-cdz7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:17.141444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.141416 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-xks-ca" pod "d8984cf1-62df-4938-a90a-2a129e9b5e2d" (UID: "d8984cf1-62df-4938-a90a-2a129e9b5e2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:17.240202 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.240158 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdz7l\" (UniqueName: \"kubernetes.io/projected/d8984cf1-62df-4938-a90a-2a129e9b5e2d-kube-api-access-cdz7l\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:17.240202 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.240197 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8984cf1-62df-4938-a90a-2a129e9b5e2d-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:17.391249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.391175 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:17.392778 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.392754 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:52:17.392847 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.392803 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:52:17.784471 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.784394 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" Oct 31 00:52:17.784471 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.784414 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj" event={"ID":"d8984cf1-62df-4938-a90a-2a129e9b5e2d","Type":"ContainerDied","Data":"11ade1a1ba6dcd22504e0bfca158338aa54a250ded5e0d8b13a66ba2d2bf08af"} Oct 31 00:52:17.784471 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:17.784465 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11ade1a1ba6dcd22504e0bfca158338aa54a250ded5e0d8b13a66ba2d2bf08af" Oct 31 00:52:18.391297 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:18.391267 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:52:18.391469 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:18.391315 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:52:19.391198 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:19.391167 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:52:19.391680 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:19.391652 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:52:20.391278 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:20.391245 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:52:20.391661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:20.391298 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:52:21.391596 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:21.391561 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:52:21.392006 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:21.391611 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:52:22.050115 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:22.050085 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg"] Oct 31 00:52:22.102445 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:22.102409 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa983f3a-qxdpg"] Oct 31 00:52:22.391576 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:22.391543 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 00:52:22.391800 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:22.391593 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 00:52:22.610320 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:22.610276 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acff01b6-dfc5-4675-9768-4c72ad52070e" path="/var/lib/kubelet/pods/acff01b6-dfc5-4675-9768-4c72ad52070e/volumes" Oct 31 00:52:23.395441 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:23.395405 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:23.395944 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:23.395792 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:23.396422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:23.396403 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:52:24.573730 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.573702 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66"] Oct 31 00:52:24.574280 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.574266 5719 cpu_manager.go:414] "containerMap: removing container" podUID="d8984cf1-62df-4938-a90a-2a129e9b5e2d" containerName="revoke-keys" Oct 31 00:52:24.574325 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.574281 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8984cf1-62df-4938-a90a-2a129e9b5e2d" containerName="revoke-keys" Oct 31 00:52:24.574375 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.574366 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8984cf1-62df-4938-a90a-2a129e9b5e2d" containerName="revoke-keys" Oct 31 00:52:24.577792 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.577773 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.633150 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.633125 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:52:24.676487 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.676254 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:52:24.676487 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.676369 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:52:24.676487 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.676460 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:52:24.684401 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.684366 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66"] Oct 31 00:52:24.729798 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.729758 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.729992 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.729854 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjdqx\" (UniqueName: \"kubernetes.io/projected/f51c0e76-2466-4ed7-951f-338440df990f-kube-api-access-pjdqx\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.730049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.730018 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.831410 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.831329 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.831560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.831433 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjdqx\" (UniqueName: \"kubernetes.io/projected/f51c0e76-2466-4ed7-951f-338440df990f-kube-api-access-pjdqx\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.831560 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.831507 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.832093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.832072 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.834235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.834213 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.890341 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.890308 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjdqx\" (UniqueName: \"kubernetes.io/projected/f51c0e76-2466-4ed7-951f-338440df990f-kube-api-access-pjdqx\") pod \"maas-api-revoke-sub-2d840f0f-gzh66\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:24.893187 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:24.893168 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:25.275594 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:25.275526 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66"] Oct 31 00:52:25.279707 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:52:25.279666 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51c0e76_2466_4ed7_951f_338440df990f.slice/crio-aaeebe3d696fb48cae77e7349ae9e6161696b200c0880246e7a75395d411ab0c WatchSource:0}: Error finding container aaeebe3d696fb48cae77e7349ae9e6161696b200c0880246e7a75395d411ab0c: Status 404 returned error can't find the container with id aaeebe3d696fb48cae77e7349ae9e6161696b200c0880246e7a75395d411ab0c Oct 31 00:52:25.868620 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:25.868588 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" event={"ID":"f51c0e76-2466-4ed7-951f-338440df990f","Type":"ContainerStarted","Data":"c79d1ee2f41e4b2cd20b69022d8924160327c945308cd043fb66ba7783fc8190"} Oct 31 00:52:25.868997 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:25.868647 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" event={"ID":"f51c0e76-2466-4ed7-951f-338440df990f","Type":"ContainerStarted","Data":"aaeebe3d696fb48cae77e7349ae9e6161696b200c0880246e7a75395d411ab0c"} Oct 31 00:52:25.982199 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:25.982118 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" podStartSLOduration=1.982103842 podStartE2EDuration="1.982103842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:25.979164359 +0000 UTC m=+1427.515306664" watchObservedRunningTime="2026-10-31 00:52:25.982103842 +0000 UTC m=+1427.518246146" Oct 31 00:52:26.876782 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:26.876743 5719 generic.go:356] "Generic (PLEG): container finished" podID="f51c0e76-2466-4ed7-951f-338440df990f" containerID="c79d1ee2f41e4b2cd20b69022d8924160327c945308cd043fb66ba7783fc8190" exitCode=0 Oct 31 00:52:26.878974 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:26.876799 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" event={"ID":"f51c0e76-2466-4ed7-951f-338440df990f","Type":"ContainerDied","Data":"c79d1ee2f41e4b2cd20b69022d8924160327c945308cd043fb66ba7783fc8190"} Oct 31 00:52:27.457461 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:27.457430 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:52:28.038713 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.038672 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:28.185457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.184558 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51c0e76-2466-4ed7-951f-338440df990f-kube-api-access-pjdqx\" (UniqueName: \"kubernetes.io/projected/f51c0e76-2466-4ed7-951f-338440df990f-kube-api-access-pjdqx\") pod \"f51c0e76-2466-4ed7-951f-338440df990f\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " Oct 31 00:52:28.185457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.184795 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-openshift-ca\") pod \"f51c0e76-2466-4ed7-951f-338440df990f\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " Oct 31 00:52:28.185457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.184817 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-xks-ca\") pod \"f51c0e76-2466-4ed7-951f-338440df990f\" (UID: \"f51c0e76-2466-4ed7-951f-338440df990f\") " Oct 31 00:52:28.185457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.185196 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-openshift-ca" pod "f51c0e76-2466-4ed7-951f-338440df990f" (UID: "f51c0e76-2466-4ed7-951f-338440df990f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:28.185457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.185385 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:28.190193 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.190120 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51c0e76-2466-4ed7-951f-338440df990f-kube-api-access-pjdqx" pod "f51c0e76-2466-4ed7-951f-338440df990f" (UID: "f51c0e76-2466-4ed7-951f-338440df990f"). InnerVolumeSpecName "kube-api-access-pjdqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:28.191507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.191486 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-xks-ca" pod "f51c0e76-2466-4ed7-951f-338440df990f" (UID: "f51c0e76-2466-4ed7-951f-338440df990f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:28.287582 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.287540 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51c0e76-2466-4ed7-951f-338440df990f-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:28.287848 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.287588 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjdqx\" (UniqueName: \"kubernetes.io/projected/f51c0e76-2466-4ed7-951f-338440df990f-kube-api-access-pjdqx\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:28.894031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.893951 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" event={"ID":"f51c0e76-2466-4ed7-951f-338440df990f","Type":"ContainerDied","Data":"aaeebe3d696fb48cae77e7349ae9e6161696b200c0880246e7a75395d411ab0c"} Oct 31 00:52:28.894031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.893994 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66" Oct 31 00:52:28.894031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:28.894020 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaeebe3d696fb48cae77e7349ae9e6161696b200c0880246e7a75395d411ab0c" Oct 31 00:52:33.009625 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:52:33.009591 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:52:35.049849 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:35.049817 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl"] Oct 31 00:52:35.097292 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:35.097258 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0066dae7-ff5nl"] Oct 31 00:52:36.603589 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:36.603556 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17057d4f-1632-4c84-ab68-9a1f12d85551" path="/var/lib/kubelet/pods/17057d4f-1632-4c84-ab68-9a1f12d85551/volumes" Oct 31 00:52:52.049672 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:52.049585 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28"] Oct 31 00:52:52.101016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:52.100980 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5bb6bf-d5c28"] Oct 31 00:52:52.605956 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:52:52.605918 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1005dc-1399-4230-b8d7-088a0912056c" path="/var/lib/kubelet/pods/bf1005dc-1399-4230-b8d7-088a0912056c/volumes" Oct 31 00:53:03.011128 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:53:03.011093 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:53:05.876574 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.876541 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh"] Oct 31 00:53:05.877155 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.877140 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f51c0e76-2466-4ed7-951f-338440df990f" containerName="revoke-keys" Oct 31 00:53:05.877199 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.877155 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51c0e76-2466-4ed7-951f-338440df990f" containerName="revoke-keys" Oct 31 00:53:05.877262 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.877253 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51c0e76-2466-4ed7-951f-338440df990f" containerName="revoke-keys" Oct 31 00:53:05.880840 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.880822 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:05.931054 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.931032 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:05.931200 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.931112 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:53:05.935735 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.935712 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh"] Oct 31 00:53:05.938725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.938676 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:05.938845 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.938824 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:05.938984 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.938969 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwcbn\" (UniqueName: \"kubernetes.io/projected/104eb118-8eaf-4aa3-8036-8d61ead2e690-kube-api-access-nwcbn\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:05.978938 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.978914 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:05.979091 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:05.979077 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:06.040383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:06.040348 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:06.040599 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:06.040443 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:06.040709 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:06.040599 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwcbn\" (UniqueName: \"kubernetes.io/projected/104eb118-8eaf-4aa3-8036-8d61ead2e690-kube-api-access-nwcbn\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:06.041082 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:06.041060 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:06.043131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:06.043113 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-xks-ca\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:06.099627 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:06.099607 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwcbn\" (UniqueName: \"kubernetes.io/projected/104eb118-8eaf-4aa3-8036-8d61ead2e690-kube-api-access-nwcbn\") pod \"maas-api-revoke-sub-780dd372-7h8sh\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:06.193630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:06.193572 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:06.568739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:06.568709 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh"] Oct 31 00:53:06.572119 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:53:06.572085 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104eb118_8eaf_4aa3_8036_8d61ead2e690.slice/crio-0919cea3071c8680db9cb18579e88d2da5bf5ab2b8ec467f222c53ce426d5d7c WatchSource:0}: Error finding container 0919cea3071c8680db9cb18579e88d2da5bf5ab2b8ec467f222c53ce426d5d7c: Status 404 returned error can't find the container with id 0919cea3071c8680db9cb18579e88d2da5bf5ab2b8ec467f222c53ce426d5d7c Oct 31 00:53:07.054032 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:07.054007 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf"] Oct 31 00:53:07.067653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:07.067627 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4185b87-6mwvf"] Oct 31 00:53:07.181969 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:07.181898 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" event={"ID":"104eb118-8eaf-4aa3-8036-8d61ead2e690","Type":"ContainerStarted","Data":"adc33bc01baa5ec3005b8934432720dc161016d5bb5342e429d2266962e33718"} Oct 31 00:53:07.181969 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:07.181950 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" event={"ID":"104eb118-8eaf-4aa3-8036-8d61ead2e690","Type":"ContainerStarted","Data":"0919cea3071c8680db9cb18579e88d2da5bf5ab2b8ec467f222c53ce426d5d7c"} Oct 31 00:53:07.302830 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:07.302785 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" podStartSLOduration=2.30276788 podStartE2EDuration="2.30276788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:07.298732664 +0000 UTC m=+1468.834874970" watchObservedRunningTime="2026-10-31 00:53:07.30276788 +0000 UTC m=+1468.838910212" Oct 31 00:53:08.190070 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:08.190034 5719 generic.go:356] "Generic (PLEG): container finished" podID="104eb118-8eaf-4aa3-8036-8d61ead2e690" containerID="adc33bc01baa5ec3005b8934432720dc161016d5bb5342e429d2266962e33718" exitCode=0 Oct 31 00:53:08.190463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:08.190125 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" event={"ID":"104eb118-8eaf-4aa3-8036-8d61ead2e690","Type":"ContainerDied","Data":"adc33bc01baa5ec3005b8934432720dc161016d5bb5342e429d2266962e33718"} Oct 31 00:53:08.605005 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:08.604973 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3616268-b555-45bb-9d19-ea73c288999d" path="/var/lib/kubelet/pods/e3616268-b555-45bb-9d19-ea73c288999d/volumes" Oct 31 00:53:09.356280 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.356253 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:09.483541 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.483466 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-openshift-ca\") pod \"104eb118-8eaf-4aa3-8036-8d61ead2e690\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " Oct 31 00:53:09.483734 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.483635 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/104eb118-8eaf-4aa3-8036-8d61ead2e690-kube-api-access-nwcbn\" (UniqueName: \"kubernetes.io/projected/104eb118-8eaf-4aa3-8036-8d61ead2e690-kube-api-access-nwcbn\") pod \"104eb118-8eaf-4aa3-8036-8d61ead2e690\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " Oct 31 00:53:09.483810 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.483785 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-openshift-ca" pod "104eb118-8eaf-4aa3-8036-8d61ead2e690" (UID: "104eb118-8eaf-4aa3-8036-8d61ead2e690"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:09.483866 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.483804 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-xks-ca\") pod \"104eb118-8eaf-4aa3-8036-8d61ead2e690\" (UID: \"104eb118-8eaf-4aa3-8036-8d61ead2e690\") " Oct 31 00:53:09.484329 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.484305 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:09.486006 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.485977 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/104eb118-8eaf-4aa3-8036-8d61ead2e690-kube-api-access-nwcbn" pod "104eb118-8eaf-4aa3-8036-8d61ead2e690" (UID: "104eb118-8eaf-4aa3-8036-8d61ead2e690"). InnerVolumeSpecName "kube-api-access-nwcbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:09.486313 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.486294 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-xks-ca" pod "104eb118-8eaf-4aa3-8036-8d61ead2e690" (UID: "104eb118-8eaf-4aa3-8036-8d61ead2e690"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:09.586201 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.586178 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwcbn\" (UniqueName: \"kubernetes.io/projected/104eb118-8eaf-4aa3-8036-8d61ead2e690-kube-api-access-nwcbn\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:09.586201 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:09.586203 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/104eb118-8eaf-4aa3-8036-8d61ead2e690-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:10.206102 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:10.206077 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" Oct 31 00:53:10.206252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:10.206114 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh" event={"ID":"104eb118-8eaf-4aa3-8036-8d61ead2e690","Type":"ContainerDied","Data":"0919cea3071c8680db9cb18579e88d2da5bf5ab2b8ec467f222c53ce426d5d7c"} Oct 31 00:53:10.206252 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:10.206165 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0919cea3071c8680db9cb18579e88d2da5bf5ab2b8ec467f222c53ce426d5d7c" Oct 31 00:53:16.027291 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.027259 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7"] Oct 31 00:53:16.027823 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.027807 5719 cpu_manager.go:414] "containerMap: removing container" podUID="104eb118-8eaf-4aa3-8036-8d61ead2e690" containerName="revoke-keys" Oct 31 00:53:16.027867 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.027824 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="104eb118-8eaf-4aa3-8036-8d61ead2e690" containerName="revoke-keys" Oct 31 00:53:16.027951 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.027942 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="104eb118-8eaf-4aa3-8036-8d61ead2e690" containerName="revoke-keys" Oct 31 00:53:16.031903 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.031887 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.086609 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.086583 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7"] Oct 31 00:53:16.129866 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.129841 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:16.130008 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.129880 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:16.130008 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.129880 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:16.130118 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.130029 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:53:16.161392 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.161365 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxkp\" (UniqueName: \"kubernetes.io/projected/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-kube-api-access-sqxkp\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.161514 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.161488 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.161584 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.161567 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.263015 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.262986 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.263136 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.263120 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxkp\" (UniqueName: \"kubernetes.io/projected/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-kube-api-access-sqxkp\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.263253 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.263236 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.263661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.263644 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.265650 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.265632 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.321808 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.321778 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxkp\" (UniqueName: \"kubernetes.io/projected/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-kube-api-access-sqxkp\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.345612 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.345593 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:16.720290 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:16.720074 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7"] Oct 31 00:53:16.723488 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:53:16.723449 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c0c3e5_eed4_4ce5_9ec2_22b5443f4db7.slice/crio-fc8f5896fcd3d3998b792ab0b5bde5d8a96f67578b9bf5faf5875386aa129614 WatchSource:0}: Error finding container fc8f5896fcd3d3998b792ab0b5bde5d8a96f67578b9bf5faf5875386aa129614: Status 404 returned error can't find the container with id fc8f5896fcd3d3998b792ab0b5bde5d8a96f67578b9bf5faf5875386aa129614 Oct 31 00:53:17.257354 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:17.257322 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" event={"ID":"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7","Type":"ContainerStarted","Data":"e8ca1b26f1506f4d01caa5073d98c2d36d76c2ce7d4091b2ded4151e6e31f89d"} Oct 31 00:53:17.257821 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:17.257382 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" event={"ID":"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7","Type":"ContainerStarted","Data":"fc8f5896fcd3d3998b792ab0b5bde5d8a96f67578b9bf5faf5875386aa129614"} Oct 31 00:53:17.371616 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:17.371564 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" podStartSLOduration=2.371551232 podStartE2EDuration="2.371551232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:17.368189565 +0000 UTC m=+1478.904331870" watchObservedRunningTime="2026-10-31 00:53:17.371551232 +0000 UTC m=+1478.907693529" Oct 31 00:53:18.244409 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:18.244376 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:53:18.268475 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:18.268397 5719 generic.go:356] "Generic (PLEG): container finished" podID="11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7" containerID="e8ca1b26f1506f4d01caa5073d98c2d36d76c2ce7d4091b2ded4151e6e31f89d" exitCode=0 Oct 31 00:53:18.268904 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:18.268484 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" event={"ID":"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7","Type":"ContainerDied","Data":"e8ca1b26f1506f4d01caa5073d98c2d36d76c2ce7d4091b2ded4151e6e31f89d"} Oct 31 00:53:19.468982 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.468958 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:19.617850 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.617815 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-xks-ca\") pod \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " Oct 31 00:53:19.618489 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.618460 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-openshift-ca\") pod \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " Oct 31 00:53:19.618638 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.618548 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-kube-api-access-sqxkp\" (UniqueName: \"kubernetes.io/projected/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-kube-api-access-sqxkp\") pod \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\" (UID: \"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7\") " Oct 31 00:53:19.619251 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.619226 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-openshift-ca" pod "11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7" (UID: "11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:19.621166 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.621136 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-xks-ca" pod "11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7" (UID: "11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:19.630359 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.630333 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-kube-api-access-sqxkp" pod "11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7" (UID: "11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7"). InnerVolumeSpecName "kube-api-access-sqxkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:19.720844 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.720808 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:19.720844 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.720838 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:19.720844 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:19.720851 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxkp\" (UniqueName: \"kubernetes.io/projected/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7-kube-api-access-sqxkp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:20.286782 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:20.286739 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" event={"ID":"11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7","Type":"ContainerDied","Data":"fc8f5896fcd3d3998b792ab0b5bde5d8a96f67578b9bf5faf5875386aa129614"} Oct 31 00:53:20.286985 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:20.286810 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc8f5896fcd3d3998b792ab0b5bde5d8a96f67578b9bf5faf5875386aa129614" Oct 31 00:53:20.286985 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:20.286750 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7" Oct 31 00:53:22.056797 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:22.056758 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt"] Oct 31 00:53:22.102269 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:22.102237 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02e2323d-sw7tt"] Oct 31 00:53:22.602745 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:22.602715 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1fdda0-ebdd-43bd-abab-722c4010b1db" path="/var/lib/kubelet/pods/5c1fdda0-ebdd-43bd-abab-722c4010b1db/volumes" Oct 31 00:53:23.049187 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:23.049111 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr"] Oct 31 00:53:23.052470 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:23.052447 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b92f53-lvldr"] Oct 31 00:53:24.604725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:24.604679 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81e02a5-6d58-4372-8a0a-3a85d0919f45" path="/var/lib/kubelet/pods/f81e02a5-6d58-4372-8a0a-3a85d0919f45/volumes" Oct 31 00:53:26.054170 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:26.054138 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml"] Oct 31 00:53:26.101912 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:26.101882 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07df7525-hlnml"] Oct 31 00:53:26.604172 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:26.604142 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22c368a-fd32-4e28-bcb8-189a25a2e3ec" path="/var/lib/kubelet/pods/b22c368a-fd32-4e28-bcb8-189a25a2e3ec/volumes" Oct 31 00:53:33.010915 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:53:33.010873 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:53:34.051402 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:34.051370 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt"] Oct 31 00:53:34.099621 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:34.099590 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-135c2f71-sv6kt"] Oct 31 00:53:34.602672 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:34.602630 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2eaf0d-d419-49ef-b25f-b30ac65745cf" path="/var/lib/kubelet/pods/1b2eaf0d-d419-49ef-b25f-b30ac65745cf/volumes" Oct 31 00:53:36.057286 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:36.057246 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq"] Oct 31 00:53:36.102914 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:36.102886 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-726b07a7-b4fsq"] Oct 31 00:53:36.604363 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:36.604328 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01760557-408a-4298-833c-6b948b61c428" path="/var/lib/kubelet/pods/01760557-408a-4298-833c-6b948b61c428/volumes" Oct 31 00:53:39.650573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.650543 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj"] Oct 31 00:53:39.651109 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.651094 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7" containerName="revoke-keys" Oct 31 00:53:39.651176 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.651110 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7" containerName="revoke-keys" Oct 31 00:53:39.651226 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.651217 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7" containerName="revoke-keys" Oct 31 00:53:39.656173 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.656154 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.707029 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.707002 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:39.710127 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.710107 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj"] Oct 31 00:53:39.752906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.752877 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:39.752906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.752883 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:53:39.752906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.752903 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:39.818807 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.818782 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.818993 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.818973 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnzdz\" (UniqueName: \"kubernetes.io/projected/08d31f9f-2c7f-403b-b569-9ee6787910f0-kube-api-access-xnzdz\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.819042 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.819006 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.920319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.920242 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.920444 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.920422 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnzdz\" (UniqueName: \"kubernetes.io/projected/08d31f9f-2c7f-403b-b569-9ee6787910f0-kube-api-access-xnzdz\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.920516 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.920469 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.921106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.921085 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.922630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.922606 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:39.979229 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:39.979210 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnzdz\" (UniqueName: \"kubernetes.io/projected/08d31f9f-2c7f-403b-b569-9ee6787910f0-kube-api-access-xnzdz\") pod \"maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:40.250513 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:40.250429 5719 scope.go:122] "RemoveContainer" containerID="43a1da746396129d0a6ba0d70c0711ca94c5552282bded90457656956b567e9a" Oct 31 00:53:40.279335 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:40.278960 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:40.279825 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:40.279657 5719 scope.go:122] "RemoveContainer" containerID="937f03afad30a438c3f887ac3897ee07e2e95d6ca5a075ec5ada2e355b9cdcf8" Oct 31 00:53:40.695832 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:40.695798 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj"] Oct 31 00:53:40.699549 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:53:40.699507 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d31f9f_2c7f_403b_b569_9ee6787910f0.slice/crio-5ba130375ce86dc0f68b2e60175ad6d1e2106faacf4ce878677628fbaf04a965 WatchSource:0}: Error finding container 5ba130375ce86dc0f68b2e60175ad6d1e2106faacf4ce878677628fbaf04a965: Status 404 returned error can't find the container with id 5ba130375ce86dc0f68b2e60175ad6d1e2106faacf4ce878677628fbaf04a965 Oct 31 00:53:41.450926 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:41.450841 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" event={"ID":"08d31f9f-2c7f-403b-b569-9ee6787910f0","Type":"ContainerStarted","Data":"6db4166722fa0ea3a36684ce59ef6aa87a24d14b3f432cd61dec4c137ccd9101"} Oct 31 00:53:41.450926 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:41.450910 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" event={"ID":"08d31f9f-2c7f-403b-b569-9ee6787910f0","Type":"ContainerStarted","Data":"5ba130375ce86dc0f68b2e60175ad6d1e2106faacf4ce878677628fbaf04a965"} Oct 31 00:53:41.564667 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:41.564619 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" podStartSLOduration=2.564606634 podStartE2EDuration="2.564606634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:41.562002679 +0000 UTC m=+1503.098144986" watchObservedRunningTime="2026-10-31 00:53:41.564606634 +0000 UTC m=+1503.100748969" Oct 31 00:53:42.458862 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:42.458826 5719 generic.go:356] "Generic (PLEG): container finished" podID="08d31f9f-2c7f-403b-b569-9ee6787910f0" containerID="6db4166722fa0ea3a36684ce59ef6aa87a24d14b3f432cd61dec4c137ccd9101" exitCode=0 Oct 31 00:53:42.459281 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:42.458906 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" event={"ID":"08d31f9f-2c7f-403b-b569-9ee6787910f0","Type":"ContainerDied","Data":"6db4166722fa0ea3a36684ce59ef6aa87a24d14b3f432cd61dec4c137ccd9101"} Oct 31 00:53:43.050183 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.050145 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b"] Oct 31 00:53:43.053619 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.053594 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987dcff7-bnf2b"] Oct 31 00:53:43.632406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.632385 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:43.787011 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.786941 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-xks-ca\") pod \"08d31f9f-2c7f-403b-b569-9ee6787910f0\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " Oct 31 00:53:43.787142 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.787058 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d31f9f-2c7f-403b-b569-9ee6787910f0-kube-api-access-xnzdz\" (UniqueName: \"kubernetes.io/projected/08d31f9f-2c7f-403b-b569-9ee6787910f0-kube-api-access-xnzdz\") pod \"08d31f9f-2c7f-403b-b569-9ee6787910f0\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " Oct 31 00:53:43.787203 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.787191 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-openshift-ca\") pod \"08d31f9f-2c7f-403b-b569-9ee6787910f0\" (UID: \"08d31f9f-2c7f-403b-b569-9ee6787910f0\") " Oct 31 00:53:43.787790 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.787747 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-openshift-ca" pod "08d31f9f-2c7f-403b-b569-9ee6787910f0" (UID: "08d31f9f-2c7f-403b-b569-9ee6787910f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:43.789154 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.789130 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-xks-ca" pod "08d31f9f-2c7f-403b-b569-9ee6787910f0" (UID: "08d31f9f-2c7f-403b-b569-9ee6787910f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:43.789242 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.789200 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d31f9f-2c7f-403b-b569-9ee6787910f0-kube-api-access-xnzdz" pod "08d31f9f-2c7f-403b-b569-9ee6787910f0" (UID: "08d31f9f-2c7f-403b-b569-9ee6787910f0"). InnerVolumeSpecName "kube-api-access-xnzdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:43.888438 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.888411 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnzdz\" (UniqueName: \"kubernetes.io/projected/08d31f9f-2c7f-403b-b569-9ee6787910f0-kube-api-access-xnzdz\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:43.888438 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.888434 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:43.888574 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:43.888444 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d31f9f-2c7f-403b-b569-9ee6787910f0-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:44.474162 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:44.474132 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" Oct 31 00:53:44.474392 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:44.474165 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj" event={"ID":"08d31f9f-2c7f-403b-b569-9ee6787910f0","Type":"ContainerDied","Data":"5ba130375ce86dc0f68b2e60175ad6d1e2106faacf4ce878677628fbaf04a965"} Oct 31 00:53:44.474392 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:44.474208 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ba130375ce86dc0f68b2e60175ad6d1e2106faacf4ce878677628fbaf04a965" Oct 31 00:53:44.605320 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:44.605281 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead3b38f-166b-4092-b6f8-5d4418411b3a" path="/var/lib/kubelet/pods/ead3b38f-166b-4092-b6f8-5d4418411b3a/volumes" Oct 31 00:53:48.872001 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.871968 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j"] Oct 31 00:53:48.872539 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.872525 5719 cpu_manager.go:414] "containerMap: removing container" podUID="08d31f9f-2c7f-403b-b569-9ee6787910f0" containerName="revoke-keys" Oct 31 00:53:48.872580 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.872541 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d31f9f-2c7f-403b-b569-9ee6787910f0" containerName="revoke-keys" Oct 31 00:53:48.872661 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.872652 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d31f9f-2c7f-403b-b569-9ee6787910f0" containerName="revoke-keys" Oct 31 00:53:48.877025 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.877003 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:48.930872 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.930848 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j"] Oct 31 00:53:48.945072 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.945048 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:48.945206 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.945185 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:48.945275 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.945257 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xwc\" (UniqueName: \"kubernetes.io/projected/89bceddb-a966-41d9-a6ec-13922f37d7c4-kube-api-access-97xwc\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:48.974775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.974750 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:53:48.974948 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.974889 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:53:48.975085 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.975062 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:53:48.975291 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:48.975240 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:53:49.046842 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:49.046819 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:49.046957 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:49.046866 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xwc\" (UniqueName: \"kubernetes.io/projected/89bceddb-a966-41d9-a6ec-13922f37d7c4-kube-api-access-97xwc\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:49.047022 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:49.046960 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:49.047458 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:49.047436 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:49.049383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:49.049360 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:49.106410 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:49.106382 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xwc\" (UniqueName: \"kubernetes.io/projected/89bceddb-a966-41d9-a6ec-13922f37d7c4-kube-api-access-97xwc\") pod \"maas-api-revoke-sub-b11c4c29-bt56j\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:49.189727 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:49.189652 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:49.565403 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:49.565323 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j"] Oct 31 00:53:50.516965 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:50.516934 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" event={"ID":"89bceddb-a966-41d9-a6ec-13922f37d7c4","Type":"ContainerStarted","Data":"3d0ec279052bd2d31f701a5665135c65c088a93b44fd3d47e2c4d7f9c3bed32f"} Oct 31 00:53:50.517315 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:50.516983 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" event={"ID":"89bceddb-a966-41d9-a6ec-13922f37d7c4","Type":"ContainerStarted","Data":"b8911bb3d0a067473f64592b626295264129433e009e7373d0ec5bce6ade7066"} Oct 31 00:53:50.639358 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:50.639306 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" podStartSLOduration=2.639288666 podStartE2EDuration="2.639288666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:50.636543048 +0000 UTC m=+1512.172685354" watchObservedRunningTime="2026-10-31 00:53:50.639288666 +0000 UTC m=+1512.175430975" Oct 31 00:53:51.531414 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:51.531383 5719 generic.go:356] "Generic (PLEG): container finished" podID="89bceddb-a966-41d9-a6ec-13922f37d7c4" containerID="3d0ec279052bd2d31f701a5665135c65c088a93b44fd3d47e2c4d7f9c3bed32f" exitCode=0 Oct 31 00:53:51.531907 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:51.531472 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" event={"ID":"89bceddb-a966-41d9-a6ec-13922f37d7c4","Type":"ContainerDied","Data":"3d0ec279052bd2d31f701a5665135c65c088a93b44fd3d47e2c4d7f9c3bed32f"} Oct 31 00:53:52.768839 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.768524 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:52.790788 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.790763 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-xks-ca\") pod \"89bceddb-a966-41d9-a6ec-13922f37d7c4\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " Oct 31 00:53:52.790917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.790797 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-openshift-ca\") pod \"89bceddb-a966-41d9-a6ec-13922f37d7c4\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " Oct 31 00:53:52.790917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.790852 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bceddb-a966-41d9-a6ec-13922f37d7c4-kube-api-access-97xwc\" (UniqueName: \"kubernetes.io/projected/89bceddb-a966-41d9-a6ec-13922f37d7c4-kube-api-access-97xwc\") pod \"89bceddb-a966-41d9-a6ec-13922f37d7c4\" (UID: \"89bceddb-a966-41d9-a6ec-13922f37d7c4\") " Oct 31 00:53:52.791168 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.791145 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-openshift-ca" pod "89bceddb-a966-41d9-a6ec-13922f37d7c4" (UID: "89bceddb-a966-41d9-a6ec-13922f37d7c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:53:52.791372 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.791359 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:52.793175 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.793156 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-xks-ca" pod "89bceddb-a966-41d9-a6ec-13922f37d7c4" (UID: "89bceddb-a966-41d9-a6ec-13922f37d7c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:53:52.793374 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.793355 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bceddb-a966-41d9-a6ec-13922f37d7c4-kube-api-access-97xwc" pod "89bceddb-a966-41d9-a6ec-13922f37d7c4" (UID: "89bceddb-a966-41d9-a6ec-13922f37d7c4"). InnerVolumeSpecName "kube-api-access-97xwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:53:52.893076 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.893037 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bceddb-a966-41d9-a6ec-13922f37d7c4-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:52.893076 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:52.893074 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97xwc\" (UniqueName: \"kubernetes.io/projected/89bceddb-a966-41d9-a6ec-13922f37d7c4-kube-api-access-97xwc\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:53:53.050428 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:53.050396 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d"] Oct 31 00:53:53.053683 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:53.053661 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ab2c482-cb44d"] Oct 31 00:53:53.560506 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:53.560479 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" Oct 31 00:53:53.560665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:53.560474 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j" event={"ID":"89bceddb-a966-41d9-a6ec-13922f37d7c4","Type":"ContainerDied","Data":"b8911bb3d0a067473f64592b626295264129433e009e7373d0ec5bce6ade7066"} Oct 31 00:53:53.560665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:53.560601 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8911bb3d0a067473f64592b626295264129433e009e7373d0ec5bce6ade7066" Oct 31 00:53:54.606570 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:53:54.606515 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5b44d5-caa1-44b4-bbe9-e77438cbd43c" path="/var/lib/kubelet/pods/1c5b44d5-caa1-44b4-bbe9-e77438cbd43c/volumes" Oct 31 00:54:02.053078 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:02.053046 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw"] Oct 31 00:54:02.099014 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:02.098986 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d1b8178-zhsrw"] Oct 31 00:54:02.603798 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:02.603766 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51a0864-ff12-4b07-9a8f-678fda32f69b" path="/var/lib/kubelet/pods/e51a0864-ff12-4b07-9a8f-678fda32f69b/volumes" Oct 31 00:54:03.010810 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:54:03.010728 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:54:05.049763 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:05.049728 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg"] Oct 31 00:54:05.097303 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:05.097275 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06c3575-fr6gg"] Oct 31 00:54:06.604150 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:06.604114 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b150081f-6e67-4690-a2f8-f3a27987376b" path="/var/lib/kubelet/pods/b150081f-6e67-4690-a2f8-f3a27987376b/volumes" Oct 31 00:54:08.049495 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:08.049460 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb"] Oct 31 00:54:08.096955 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:08.096927 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e1e00a7-jfxcb"] Oct 31 00:54:08.612795 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:08.612755 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1bddbf-0480-4544-8efa-342c145cd2cc" path="/var/lib/kubelet/pods/5a1bddbf-0480-4544-8efa-342c145cd2cc/volumes" Oct 31 00:54:15.598633 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:15.598558 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:54:21.047997 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:21.047962 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9"] Oct 31 00:54:21.097407 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:21.097381 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41499fca-v94j9"] Oct 31 00:54:22.611749 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:22.611709 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8582bf9-7d29-473f-9274-260270f96b96" path="/var/lib/kubelet/pods/b8582bf9-7d29-473f-9274-260270f96b96/volumes" Oct 31 00:54:28.049583 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:28.049543 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm"] Oct 31 00:54:28.096968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:28.096937 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa833666-9mtqm"] Oct 31 00:54:28.604084 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:28.604053 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3712b2fb-38cf-4576-a36e-ef6235104a7a" path="/var/lib/kubelet/pods/3712b2fb-38cf-4576-a36e-ef6235104a7a/volumes" Oct 31 00:54:33.010557 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:54:33.010515 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:54:38.052795 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:38.052759 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x"] Oct 31 00:54:38.112219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:38.112186 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcf4984c-pg82x"] Oct 31 00:54:38.606317 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:38.606283 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46a6850-515a-4ced-a053-75c4ec059a56" path="/var/lib/kubelet/pods/f46a6850-515a-4ced-a053-75c4ec059a56/volumes" Oct 31 00:54:40.349158 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.349124 5719 scope.go:122] "RemoveContainer" containerID="945c11e99d491ddeb07e07131d9cc67b750856286bbfd49d461bb105bb99bf3a" Oct 31 00:54:40.359412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.359392 5719 scope.go:122] "RemoveContainer" containerID="af0b1d1e8c2505bb1f50ec3735108a30d790cb10d9c9f5eb3eba2316d2615097" Oct 31 00:54:40.368491 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.368473 5719 scope.go:122] "RemoveContainer" containerID="b9b6887db39639f25e95618cbfc6bb8fbdcb6431b5b09a87679514b7420e4a61" Oct 31 00:54:40.380242 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.380216 5719 scope.go:122] "RemoveContainer" containerID="1d607b10af1a317770b757a62d32c855808b3c529a0f3244c663c8fae66f98d0" Oct 31 00:54:40.388521 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.388500 5719 scope.go:122] "RemoveContainer" containerID="f5d835e456720c170166917d965f4d8b5385105f47bb658b734a2d8422c33910" Oct 31 00:54:40.396989 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.396969 5719 scope.go:122] "RemoveContainer" containerID="5635bdab22f5a547512d9a0b19e17b49d5287261472c58a3397b4b65640929d1" Oct 31 00:54:40.404918 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.404902 5719 scope.go:122] "RemoveContainer" containerID="57928598a04364d4082cccdfaa29034a14eb97c73625fe428f84f576a13688b6" Oct 31 00:54:40.413054 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.413038 5719 scope.go:122] "RemoveContainer" containerID="9c3e5684e27f0d2488e9d2ebd03acf6a7be64ce795e627fa9c7548a85d52290e" Oct 31 00:54:40.453352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.453331 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7"] Oct 31 00:54:40.500656 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.500629 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-b6rr7"] Oct 31 00:54:40.608314 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:40.608239 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7" path="/var/lib/kubelet/pods/11c0c3e5-eed4-4ce5-9ec2-22b5443f4db7/volumes" Oct 31 00:54:41.047340 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:41.047265 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc"] Oct 31 00:54:41.094840 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:41.094815 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-509f1050-fbkbc"] Oct 31 00:54:42.604977 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:42.604944 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7f695e-4522-48cf-8a15-9c0e6a412d37" path="/var/lib/kubelet/pods/bc7f695e-4522-48cf-8a15-9c0e6a412d37/volumes" Oct 31 00:54:46.810369 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:46.810338 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj"] Oct 31 00:54:46.857526 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:46.857493 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8c2357-mc8wj"] Oct 31 00:54:48.602406 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:48.602375 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d31f9f-2c7f-403b-b569-9ee6787910f0" path="/var/lib/kubelet/pods/08d31f9f-2c7f-403b-b569-9ee6787910f0/volumes" Oct 31 00:54:56.551423 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.551388 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v"] Oct 31 00:54:56.551952 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.551939 5719 cpu_manager.go:414] "containerMap: removing container" podUID="89bceddb-a966-41d9-a6ec-13922f37d7c4" containerName="revoke-keys" Oct 31 00:54:56.552011 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.551953 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89bceddb-a966-41d9-a6ec-13922f37d7c4" containerName="revoke-keys" Oct 31 00:54:56.552107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.552097 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="89bceddb-a966-41d9-a6ec-13922f37d7c4" containerName="revoke-keys" Oct 31 00:54:56.555351 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.555335 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.606856 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.606820 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:54:56.654668 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.654640 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:54:56.654668 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.654655 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:54:56.654852 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.654677 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:54:56.657685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.657663 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v"] Oct 31 00:54:56.696132 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.696111 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwldj\" (UniqueName: \"kubernetes.io/projected/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-kube-api-access-lwldj\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.696224 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.696164 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.696261 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.696248 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.797330 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.797304 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwldj\" (UniqueName: \"kubernetes.io/projected/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-kube-api-access-lwldj\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.797426 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.797362 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.797426 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.797392 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.798016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.797997 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.799773 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.799756 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.856292 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.856271 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwldj\" (UniqueName: \"kubernetes.io/projected/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-kube-api-access-lwldj\") pod \"maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:56.868952 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:56.868933 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:54:57.245287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:57.245197 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v"] Oct 31 00:54:57.249302 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:54:57.249274 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ae4c9b_e9ee_4b59_b3bd_fa2e0634493e.slice/crio-9dbc6848326740ecba45693aace83f3957126020d0b7ea5a84067861cea2673a WatchSource:0}: Error finding container 9dbc6848326740ecba45693aace83f3957126020d0b7ea5a84067861cea2673a: Status 404 returned error can't find the container with id 9dbc6848326740ecba45693aace83f3957126020d0b7ea5a84067861cea2673a Oct 31 00:54:58.066054 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:58.066019 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" event={"ID":"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e","Type":"ContainerStarted","Data":"0c96304160b8908ccbd1464f34735258af78cbf9293f23a125fc4a1361a4d64e"} Oct 31 00:54:58.066453 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:58.066081 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" event={"ID":"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e","Type":"ContainerStarted","Data":"9dbc6848326740ecba45693aace83f3957126020d0b7ea5a84067861cea2673a"} Oct 31 00:54:58.186677 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:58.186627 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" podStartSLOduration=2.186614838 podStartE2EDuration="2.186614838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:58.185104783 +0000 UTC m=+1579.721247088" watchObservedRunningTime="2026-10-31 00:54:58.186614838 +0000 UTC m=+1579.722757142" Oct 31 00:54:59.074072 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:59.074034 5719 generic.go:356] "Generic (PLEG): container finished" podID="28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e" containerID="0c96304160b8908ccbd1464f34735258af78cbf9293f23a125fc4a1361a4d64e" exitCode=0 Oct 31 00:54:59.074441 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:59.074120 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" event={"ID":"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e","Type":"ContainerDied","Data":"0c96304160b8908ccbd1464f34735258af78cbf9293f23a125fc4a1361a4d64e"} Oct 31 00:54:59.857016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:59.856980 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:54:59.940258 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:59.940228 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6"] Oct 31 00:54:59.940723 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:59.940624 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" containerID="cri-o://cca709f1b8f75da9891b8505c07fee59f01bdd077e7fdc51cb3fda74c84ec2a3" gracePeriod=30 Oct 31 00:54:59.945110 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:59.945088 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4"] Oct 31 00:54:59.945285 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:54:59.945262 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" containerID="cri-o://212350ae167789d808d67663abe422e14c7744a2649b0abcb99e45d0a7bb3d95" gracePeriod=30 Oct 31 00:55:00.241538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.241512 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:55:00.241933 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.241878 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-openshift-ca\") pod \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " Oct 31 00:55:00.242016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.241974 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-kube-api-access-lwldj\" (UniqueName: \"kubernetes.io/projected/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-kube-api-access-lwldj\") pod \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " Oct 31 00:55:00.242016 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.242012 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-xks-ca\") pod \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\" (UID: \"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e\") " Oct 31 00:55:00.242226 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.242204 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-openshift-ca" pod "28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e" (UID: "28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:00.242568 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.242542 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:00.244136 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.244113 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-kube-api-access-lwldj" pod "28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e" (UID: "28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e"). InnerVolumeSpecName "kube-api-access-lwldj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:00.244269 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.244254 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-xks-ca" pod "28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e" (UID: "28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:00.343468 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.343433 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:00.343468 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:00.343466 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwldj\" (UniqueName: \"kubernetes.io/projected/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e-kube-api-access-lwldj\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:01.100797 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:01.096728 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" event={"ID":"28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e","Type":"ContainerDied","Data":"9dbc6848326740ecba45693aace83f3957126020d0b7ea5a84067861cea2673a"} Oct 31 00:55:01.100797 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:01.096796 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dbc6848326740ecba45693aace83f3957126020d0b7ea5a84067861cea2673a" Oct 31 00:55:01.100797 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:01.096867 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v" Oct 31 00:55:03.010511 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:55:03.010473 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:55:05.131522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.131332 5719 generic.go:356] "Generic (PLEG): container finished" podID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerID="212350ae167789d808d67663abe422e14c7744a2649b0abcb99e45d0a7bb3d95" exitCode=0 Oct 31 00:55:05.131522 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.131459 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" event={"ID":"05e9f4cf-d892-472c-9504-ed0c33ff8fff","Type":"ContainerDied","Data":"212350ae167789d808d67663abe422e14c7744a2649b0abcb99e45d0a7bb3d95"} Oct 31 00:55:05.134123 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.134097 5719 generic.go:356] "Generic (PLEG): container finished" podID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerID="cca709f1b8f75da9891b8505c07fee59f01bdd077e7fdc51cb3fda74c84ec2a3" exitCode=0 Oct 31 00:55:05.134239 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.134222 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" event={"ID":"34b72ba3-cf95-40c0-a436-31921b9a72d9","Type":"ContainerDied","Data":"cca709f1b8f75da9891b8505c07fee59f01bdd077e7fdc51cb3fda74c84ec2a3"} Oct 31 00:55:05.219500 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.219444 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:55:05.254747 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.254724 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:55:05.321268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321243 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-token\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-token\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.321382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321273 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-podinfo\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.321382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321313 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-credential-socket\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.321496 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321447 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-socket\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.321554 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321491 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-token\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-token\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.321554 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321497 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-credential-socket" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.321654 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321552 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-credential-socket\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.321654 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321602 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-kube-api-access-r8kcm\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-kube-api-access-r8kcm\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.321793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321659 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-socket" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.321793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321732 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b72ba3-cf95-40c0-a436-31921b9a72d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b72ba3-cf95-40c0-a436-31921b9a72d9-istiod-ca-cert\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.321793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321772 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-certs\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.321936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321803 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-kube-api-access-xpwt9\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-kube-api-access-xpwt9\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.322114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321893 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-credential-socket" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.322114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.321979 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-podinfo\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.322238 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322138 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-certs" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.322238 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322146 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-data\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.322354 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322328 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-data" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.322414 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322379 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-socket\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.322471 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322430 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istiod-ca-cert\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.322471 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322460 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-envoy\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.322770 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322504 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-certs\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.322770 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322539 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-envoy\") pod \"34b72ba3-cf95-40c0-a436-31921b9a72d9\" (UID: \"34b72ba3-cf95-40c0-a436-31921b9a72d9\") " Oct 31 00:55:05.322770 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322581 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-data\") pod \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\" (UID: \"05e9f4cf-d892-472c-9504-ed0c33ff8fff\") " Oct 31 00:55:05.322944 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.322866 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b72ba3-cf95-40c0-a436-31921b9a72d9-istiod-ca-cert" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:05.324068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.323739 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.324068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.323765 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.324068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.323779 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.324068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.323795 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.324068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.323808 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b72ba3-cf95-40c0-a436-31921b9a72d9-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.324068 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.323821 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.324452 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324182 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-kube-api-access-r8kcm" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "kube-api-access-r8kcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:05.324452 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324212 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-podinfo" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:05.324452 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324264 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-token" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:05.324452 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324413 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-socket" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.324629 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324482 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-certs" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.324629 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324512 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-data" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.324629 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324616 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istiod-ca-cert" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:05.324801 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324631 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-podinfo" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:05.324801 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324672 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-token" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:05.324801 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.324728 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-kube-api-access-xpwt9" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "kube-api-access-xpwt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:05.325139 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.325119 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-envoy" pod "05e9f4cf-d892-472c-9504-ed0c33ff8fff" (UID: "05e9f4cf-d892-472c-9504-ed0c33ff8fff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.326266 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.326247 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-envoy" pod "34b72ba3-cf95-40c0-a436-31921b9a72d9" (UID: "34b72ba3-cf95-40c0-a436-31921b9a72d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:05.425304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425275 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425303 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425304 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425312 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425322 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425332 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425342 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425351 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8kcm\" (UniqueName: \"kubernetes.io/projected/05e9f4cf-d892-472c-9504-ed0c33ff8fff-kube-api-access-r8kcm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425361 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwt9\" (UniqueName: \"kubernetes.io/projected/34b72ba3-cf95-40c0-a436-31921b9a72d9-kube-api-access-xpwt9\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425370 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b72ba3-cf95-40c0-a436-31921b9a72d9-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425380 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b72ba3-cf95-40c0-a436-31921b9a72d9-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425391 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:05.425525 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:05.425400 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05e9f4cf-d892-472c-9504-ed0c33ff8fff-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:06.142829 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.142804 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" Oct 31 00:55:06.143287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.142842 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4" event={"ID":"05e9f4cf-d892-472c-9504-ed0c33ff8fff","Type":"ContainerDied","Data":"82cc22dbe9c64deb18c4998cf6dda389dd8802f26d6c12cbf24805a4493a0962"} Oct 31 00:55:06.143287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.142909 5719 scope.go:122] "RemoveContainer" containerID="212350ae167789d808d67663abe422e14c7744a2649b0abcb99e45d0a7bb3d95" Oct 31 00:55:06.144567 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.144551 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" Oct 31 00:55:06.144653 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.144546 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6" event={"ID":"34b72ba3-cf95-40c0-a436-31921b9a72d9","Type":"ContainerDied","Data":"07a8cea45a8e18ad478cb9be318cae67813831c8b86585396ef67575c496e14c"} Oct 31 00:55:06.153773 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.153751 5719 scope.go:122] "RemoveContainer" containerID="cca709f1b8f75da9891b8505c07fee59f01bdd077e7fdc51cb3fda74c84ec2a3" Oct 31 00:55:06.304223 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.303856 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4"] Oct 31 00:55:06.352288 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.352260 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-t9js4"] Oct 31 00:55:06.422484 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.422450 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6"] Oct 31 00:55:06.430808 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.430779 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-hfpv6"] Oct 31 00:55:06.603249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.603214 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" path="/var/lib/kubelet/pods/05e9f4cf-d892-472c-9504-ed0c33ff8fff/volumes" Oct 31 00:55:06.603880 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:06.603865 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" path="/var/lib/kubelet/pods/34b72ba3-cf95-40c0-a436-31921b9a72d9/volumes" Oct 31 00:55:07.100948 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:07.100913 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv"] Oct 31 00:55:07.113320 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:07.113295 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523a6550-pz6tv"] Oct 31 00:55:08.606079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:08.606045 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e444959-9adb-427a-a31a-2856063afe69" path="/var/lib/kubelet/pods/4e444959-9adb-427a-a31a-2856063afe69/volumes" Oct 31 00:55:09.875788 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:09.875752 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd"] Oct 31 00:55:09.876954 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:09.875959 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" containerID="cri-o://a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e" gracePeriod=30 Oct 31 00:55:15.140583 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.140523 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:55:15.207860 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.207831 5719 generic.go:356] "Generic (PLEG): container finished" podID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerID="a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e" exitCode=0 Oct 31 00:55:15.207978 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.207876 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" Oct 31 00:55:15.207978 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.207913 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" event={"ID":"1b09fe53-f6c6-4487-9470-68d796bdf847","Type":"ContainerDied","Data":"a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e"} Oct 31 00:55:15.207978 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.207964 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd" event={"ID":"1b09fe53-f6c6-4487-9470-68d796bdf847","Type":"ContainerDied","Data":"6aebd35430fb6d9225b0775e5f98c50d9c74d364e1f201d204dcde2c3313d9d2"} Oct 31 00:55:15.208083 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.207996 5719 scope.go:122] "RemoveContainer" containerID="a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e" Oct 31 00:55:15.217553 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.217537 5719 scope.go:122] "RemoveContainer" containerID="a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e" Oct 31 00:55:15.217816 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:55:15.217799 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e\": container with ID starting with a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e not found: ID does not exist" containerID="a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e" Oct 31 00:55:15.217879 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.217825 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e"} err="failed to get container status \"a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e\": rpc error: code = NotFound desc = could not find container \"a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e\": container with ID starting with a32197dcf530cee2ff15c1ddeafd56bd9c2ba68f66adbcb5288de4ca5488226e not found: ID does not exist" Oct 31 00:55:15.250101 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250081 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b09fe53-f6c6-4487-9470-68d796bdf847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b09fe53-f6c6-4487-9470-68d796bdf847-istiod-ca-cert\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250131 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-data\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250153 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-socket\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250171 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-credential-socket\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250333 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250218 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-certs\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250333 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250242 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-kube-api-access-v2zzr\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-kube-api-access-v2zzr\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250333 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-token\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-token\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250355 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-envoy\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250397 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-podinfo\") pod \"1b09fe53-f6c6-4487-9470-68d796bdf847\" (UID: \"1b09fe53-f6c6-4487-9470-68d796bdf847\") " Oct 31 00:55:15.250573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250467 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-data" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.250573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250498 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-socket" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.250573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250522 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-certs" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.250573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250531 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b09fe53-f6c6-4487-9470-68d796bdf847-istiod-ca-cert" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:15.250852 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250808 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-credential-socket" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.250979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250963 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b09fe53-f6c6-4487-9470-68d796bdf847-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.251212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250980 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.251212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.250992 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.251212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.251001 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.251212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.251009 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.252662 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.252640 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-token" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:15.252770 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.252672 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-kube-api-access-v2zzr" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "kube-api-access-v2zzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:15.253082 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.253064 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-podinfo" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:15.253121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.253076 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-envoy" pod "1b09fe53-f6c6-4487-9470-68d796bdf847" (UID: "1b09fe53-f6c6-4487-9470-68d796bdf847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:15.353135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.353099 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2zzr\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-kube-api-access-v2zzr\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.353135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.353134 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.353319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.353152 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.353319 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.353166 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b09fe53-f6c6-4487-9470-68d796bdf847-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:15.627841 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.627810 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd"] Oct 31 00:55:15.635931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:15.635904 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-g2cpd"] Oct 31 00:55:16.276472 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.276444 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9"] Oct 31 00:55:16.276999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.276976 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" Oct 31 00:55:16.276999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.276988 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" Oct 31 00:55:16.276999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.276999 5719 cpu_manager.go:414] "containerMap: removing container" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" Oct 31 00:55:16.277107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277003 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" Oct 31 00:55:16.277107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277011 5719 cpu_manager.go:414] "containerMap: removing container" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" Oct 31 00:55:16.277107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277017 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" Oct 31 00:55:16.277107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277055 5719 cpu_manager.go:414] "containerMap: removing container" podUID="28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e" containerName="revoke-keys" Oct 31 00:55:16.277107 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277059 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e" containerName="revoke-keys" Oct 31 00:55:16.277249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277132 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" containerName="istio-proxy" Oct 31 00:55:16.277249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277139 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="05e9f4cf-d892-472c-9504-ed0c33ff8fff" containerName="istio-proxy" Oct 31 00:55:16.277249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277146 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b72ba3-cf95-40c0-a436-31921b9a72d9" containerName="istio-proxy" Oct 31 00:55:16.277249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.277155 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e" containerName="revoke-keys" Oct 31 00:55:16.281629 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.281613 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.332470 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.332420 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:55:16.332667 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.332643 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:55:16.332812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.332675 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:55:16.332812 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.332747 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:55:16.371008 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.370982 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.371133 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.371038 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9br79\" (UniqueName: \"kubernetes.io/projected/1741a680-9746-480e-b52e-e1d7ce3d4ae6-kube-api-access-9br79\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.371238 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.371226 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.382837 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.382813 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9"] Oct 31 00:55:16.472954 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.472925 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.473197 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.473178 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.473261 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.473244 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9br79\" (UniqueName: \"kubernetes.io/projected/1741a680-9746-480e-b52e-e1d7ce3d4ae6-kube-api-access-9br79\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.473748 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.473728 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.475557 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.475538 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.535454 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.535399 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9br79\" (UniqueName: \"kubernetes.io/projected/1741a680-9746-480e-b52e-e1d7ce3d4ae6-kube-api-access-9br79\") pod \"maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.594815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.594794 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:16.605371 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.605344 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b09fe53-f6c6-4487-9470-68d796bdf847" path="/var/lib/kubelet/pods/1b09fe53-f6c6-4487-9470-68d796bdf847/volumes" Oct 31 00:55:16.979387 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:16.979355 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9"] Oct 31 00:55:16.979705 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:55:16.979668 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1741a680_9746_480e_b52e_e1d7ce3d4ae6.slice/crio-c720cc256578f861d19d70a364b57cbd1478c9feba6c6fa117d5ba3b8bb72460 WatchSource:0}: Error finding container c720cc256578f861d19d70a364b57cbd1478c9feba6c6fa117d5ba3b8bb72460: Status 404 returned error can't find the container with id c720cc256578f861d19d70a364b57cbd1478c9feba6c6fa117d5ba3b8bb72460 Oct 31 00:55:17.224283 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:17.224247 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" event={"ID":"1741a680-9746-480e-b52e-e1d7ce3d4ae6","Type":"ContainerStarted","Data":"c720cc256578f861d19d70a364b57cbd1478c9feba6c6fa117d5ba3b8bb72460"} Oct 31 00:55:18.232868 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:18.232792 5719 generic.go:356] "Generic (PLEG): container finished" podID="1741a680-9746-480e-b52e-e1d7ce3d4ae6" containerID="51989ae0965a1e49795c11a6a2114b7e0e18cf32618ce3682cced4a46b33ecae" exitCode=0 Oct 31 00:55:18.233226 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:18.232873 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" event={"ID":"1741a680-9746-480e-b52e-e1d7ce3d4ae6","Type":"ContainerDied","Data":"51989ae0965a1e49795c11a6a2114b7e0e18cf32618ce3682cced4a46b33ecae"} Oct 31 00:55:19.394928 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.394900 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:19.510218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.510139 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-openshift-ca\") pod \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " Oct 31 00:55:19.510347 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.510275 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-xks-ca\") pod \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " Oct 31 00:55:19.510347 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.510319 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1741a680-9746-480e-b52e-e1d7ce3d4ae6-kube-api-access-9br79\" (UniqueName: \"kubernetes.io/projected/1741a680-9746-480e-b52e-e1d7ce3d4ae6-kube-api-access-9br79\") pod \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\" (UID: \"1741a680-9746-480e-b52e-e1d7ce3d4ae6\") " Oct 31 00:55:19.510508 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.510486 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-openshift-ca" pod "1741a680-9746-480e-b52e-e1d7ce3d4ae6" (UID: "1741a680-9746-480e-b52e-e1d7ce3d4ae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:19.510775 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.510755 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:19.512639 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.512615 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-xks-ca" pod "1741a680-9746-480e-b52e-e1d7ce3d4ae6" (UID: "1741a680-9746-480e-b52e-e1d7ce3d4ae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:55:19.512975 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.512953 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1741a680-9746-480e-b52e-e1d7ce3d4ae6-kube-api-access-9br79" pod "1741a680-9746-480e-b52e-e1d7ce3d4ae6" (UID: "1741a680-9746-480e-b52e-e1d7ce3d4ae6"). InnerVolumeSpecName "kube-api-access-9br79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:19.611507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.611483 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1741a680-9746-480e-b52e-e1d7ce3d4ae6-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:19.611507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:19.611509 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9br79\" (UniqueName: \"kubernetes.io/projected/1741a680-9746-480e-b52e-e1d7ce3d4ae6-kube-api-access-9br79\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:20.248375 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:20.248345 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" Oct 31 00:55:20.248542 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:20.248382 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9" event={"ID":"1741a680-9746-480e-b52e-e1d7ce3d4ae6","Type":"ContainerDied","Data":"c720cc256578f861d19d70a364b57cbd1478c9feba6c6fa117d5ba3b8bb72460"} Oct 31 00:55:20.248542 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:20.248430 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c720cc256578f861d19d70a364b57cbd1478c9feba6c6fa117d5ba3b8bb72460" Oct 31 00:55:33.009789 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:55:33.009753 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:55:40.272088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.272057 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:55:40.470765 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.470738 5719 scope.go:122] "RemoveContainer" containerID="c43c20f0720afc5539bcaf4731aebbab31314f2f75c111a868a1c8b641854672" Oct 31 00:55:40.480017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.480000 5719 scope.go:122] "RemoveContainer" containerID="a2644db996590c0a4e46f73ee7bb611b68f396fb6aef3bb9d5a3e295fac35d32" Oct 31 00:55:40.488457 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.488438 5719 scope.go:122] "RemoveContainer" containerID="7e27711ba5fb6f95c315b44a2fec5980453475f9ac47510f60c2cb15c0d7a244" Oct 31 00:55:40.496716 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.496683 5719 scope.go:122] "RemoveContainer" containerID="67081aa881dc2f3ae7405aa7e92c1c0b2e7045659afe0b826f2870a5e2961b95" Oct 31 00:55:40.508113 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.508098 5719 scope.go:122] "RemoveContainer" containerID="d0a5a4b72dee6dbe099e2406bb911ec6feaf271c6856354a4a95028c91f63be6" Oct 31 00:55:40.516585 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.516557 5719 scope.go:122] "RemoveContainer" containerID="2d1c79122ce759220909d2a56dea6cba16e146b1c3fddbbae3e2b59fc4a91ee4" Oct 31 00:55:40.524909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.524886 5719 scope.go:122] "RemoveContainer" containerID="2bfb657d151ef239667e339617de1a54d4f088f9602bfd88ac34aee5c2ebff03" Oct 31 00:55:40.533333 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:40.533318 5719 scope.go:122] "RemoveContainer" containerID="bcbaea7b6c71c0bac71bf1edde5e78c527f42136d301c6f48fa281e2fff78939" Oct 31 00:55:52.396083 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:52.396010 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng"] Oct 31 00:55:52.396613 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:52.396217 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" containerID="cri-o://990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383" gracePeriod=30 Oct 31 00:55:57.670791 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.670731 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:55:57.744874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.744848 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-token\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-token\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745056 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.744899 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-data\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745056 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.744953 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-podinfo\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745056 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745026 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-socket\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745056 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745052 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istiod-ca-cert\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745080 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-certs\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745113 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-credential-socket\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745202 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-envoy\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745239 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-kube-api-access-2nhtw\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-kube-api-access-2nhtw\") pod \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\" (UID: \"3468d8a4-e84e-4c45-8e70-6433e0d42da6\") " Oct 31 00:55:57.745287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745261 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-data" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.745504 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745464 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-socket" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.745542 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745521 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-certs" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.745583 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745561 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istiod-ca-cert" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:55:57.745851 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745832 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.745968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745853 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.745968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745867 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.745968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745887 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.745968 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.745954 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-credential-socket" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.747342 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.747319 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-podinfo" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:55:57.747447 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.747368 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-token" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:57.747652 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.747634 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-envoy" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:55:57.747906 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.747886 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-kube-api-access-2nhtw" pod "3468d8a4-e84e-4c45-8e70-6433e0d42da6" (UID: "3468d8a4-e84e-4c45-8e70-6433e0d42da6"). InnerVolumeSpecName "kube-api-access-2nhtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:55:57.847331 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.847301 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.847331 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.847327 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.847331 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.847337 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nhtw\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-kube-api-access-2nhtw\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.847624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.847345 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:57.847624 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:57.847357 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3468d8a4-e84e-4c45-8e70-6433e0d42da6-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:55:58.488190 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.488159 5719 generic.go:356] "Generic (PLEG): container finished" podID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerID="990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383" exitCode=0 Oct 31 00:55:58.488415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.488206 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" Oct 31 00:55:58.488415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.488219 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" event={"ID":"3468d8a4-e84e-4c45-8e70-6433e0d42da6","Type":"ContainerDied","Data":"990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383"} Oct 31 00:55:58.488415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.488261 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng" event={"ID":"3468d8a4-e84e-4c45-8e70-6433e0d42da6","Type":"ContainerDied","Data":"75376698706b4da7850c4e832a1783d6c4fd5cdd9ba8c95e7f9d7b14713e53c4"} Oct 31 00:55:58.488415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.488283 5719 scope.go:122] "RemoveContainer" containerID="990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383" Oct 31 00:55:58.499159 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.499137 5719 scope.go:122] "RemoveContainer" containerID="990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383" Oct 31 00:55:58.499427 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:55:58.499406 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383\": container with ID starting with 990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383 not found: ID does not exist" containerID="990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383" Oct 31 00:55:58.499499 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.499440 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383"} err="failed to get container status \"990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383\": rpc error: code = NotFound desc = could not find container \"990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383\": container with ID starting with 990bf35707fbdad1da26f475d794a96dd11ff0b770b41a917b90d26d06f35383 not found: ID does not exist" Oct 31 00:55:58.695847 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.695813 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng"] Oct 31 00:55:58.743607 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:55:58.743529 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-cddng"] Oct 31 00:56:00.053271 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:00.053235 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q"] Oct 31 00:56:00.053685 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:00.053482 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" containerID="cri-o://3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e" gracePeriod=30 Oct 31 00:56:00.603200 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:00.603165 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" path="/var/lib/kubelet/pods/3468d8a4-e84e-4c45-8e70-6433e0d42da6/volumes" Oct 31 00:56:03.010466 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:56:03.010423 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:56:05.319079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.319016 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:56:05.441719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.441676 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-podinfo\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.441897 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.441767 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-certs\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.441897 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.441798 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-kube-api-access-f7kx2\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.441897 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.441887 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8784813-29fc-46ba-aa14-cc965d025f0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8784813-29fc-46ba-aa14-cc965d025f0f-istiod-ca-cert\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.442121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.441910 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-socket\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.442121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.441957 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-envoy\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.442121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442037 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-credential-socket\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.442121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442065 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-data\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.442121 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442122 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-token\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-token\") pod \"c8784813-29fc-46ba-aa14-cc965d025f0f\" (UID: \"c8784813-29fc-46ba-aa14-cc965d025f0f\") " Oct 31 00:56:05.442411 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442058 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-certs" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.442411 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442379 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-credential-socket" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.442515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442412 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8784813-29fc-46ba-aa14-cc965d025f0f-istiod-ca-cert" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:05.442515 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442447 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-socket" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.442615 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442560 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-data" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.442800 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442779 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.442858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442807 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8784813-29fc-46ba-aa14-cc965d025f0f-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.442858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442822 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.442858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442837 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.442858 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.442853 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.443933 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.443912 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-podinfo" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:56:05.444529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.444511 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-token" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.444635 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.444613 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-kube-api-access-f7kx2" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "kube-api-access-f7kx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:05.444706 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.444640 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-envoy" pod "c8784813-29fc-46ba-aa14-cc965d025f0f" (UID: "c8784813-29fc-46ba-aa14-cc965d025f0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:56:05.534888 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.534854 5719 generic.go:356] "Generic (PLEG): container finished" podID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerID="3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e" exitCode=0 Oct 31 00:56:05.535088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.534902 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" Oct 31 00:56:05.535088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.534953 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" event={"ID":"c8784813-29fc-46ba-aa14-cc965d025f0f","Type":"ContainerDied","Data":"3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e"} Oct 31 00:56:05.535088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.535009 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q" event={"ID":"c8784813-29fc-46ba-aa14-cc965d025f0f","Type":"ContainerDied","Data":"88bba319697acd7b86069eccd346b5399ec444e8a5b2221b7d29f8ca1a66e550"} Oct 31 00:56:05.535088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.535038 5719 scope.go:122] "RemoveContainer" containerID="3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e" Oct 31 00:56:05.543861 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.543833 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7kx2\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-kube-api-access-f7kx2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.543962 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.543867 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.543962 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.543883 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.543962 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.543896 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8784813-29fc-46ba-aa14-cc965d025f0f-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:05.547091 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.547071 5719 scope.go:122] "RemoveContainer" containerID="3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e" Oct 31 00:56:05.547378 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:56:05.547357 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e\": container with ID starting with 3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e not found: ID does not exist" containerID="3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e" Oct 31 00:56:05.547437 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.547388 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e"} err="failed to get container status \"3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e\": rpc error: code = NotFound desc = could not find container \"3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e\": container with ID starting with 3f09562201fd515f56c6d2f93e1430d3aae8424e6f50227d684d65b53d5c588e not found: ID does not exist" Oct 31 00:56:05.696742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.696650 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q"] Oct 31 00:56:05.705135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:05.705107 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7jc7q"] Oct 31 00:56:06.603652 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:06.603616 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" path="/var/lib/kubelet/pods/c8784813-29fc-46ba-aa14-cc965d025f0f/volumes" Oct 31 00:56:19.049425 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:19.049388 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp"] Oct 31 00:56:19.096680 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:19.096650 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e29508e6-p8zcp"] Oct 31 00:56:20.604031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:20.604000 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6496ae91-5a84-451c-8e8b-9f21adf6f302" path="/var/lib/kubelet/pods/6496ae91-5a84-451c-8e8b-9f21adf6f302/volumes" Oct 31 00:56:28.916380 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:28.916348 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:56:33.012832 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:56:33.012793 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:56:34.919797 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.919767 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm"] Oct 31 00:56:34.920533 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920510 5719 cpu_manager.go:414] "containerMap: removing container" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" Oct 31 00:56:34.920533 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920532 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" Oct 31 00:56:34.920725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920559 5719 cpu_manager.go:414] "containerMap: removing container" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" Oct 31 00:56:34.920725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920567 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" Oct 31 00:56:34.920725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920582 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1741a680-9746-480e-b52e-e1d7ce3d4ae6" containerName="revoke-keys" Oct 31 00:56:34.920725 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920590 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1741a680-9746-480e-b52e-e1d7ce3d4ae6" containerName="revoke-keys" Oct 31 00:56:34.920924 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920747 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="3468d8a4-e84e-4c45-8e70-6433e0d42da6" containerName="istio-proxy" Oct 31 00:56:34.920924 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920765 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1741a680-9746-480e-b52e-e1d7ce3d4ae6" containerName="revoke-keys" Oct 31 00:56:34.920924 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.920784 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8784813-29fc-46ba-aa14-cc965d025f0f" containerName="istio-proxy" Oct 31 00:56:34.924324 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:34.924302 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.007088 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.007052 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm"] Oct 31 00:56:35.056176 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.056149 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7e6fe88b-openshift-default-dockercfg-c2g5f\"" Oct 31 00:56:35.089838 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.089802 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.089998 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.089915 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svh5s\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-kube-api-access-svh5s\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.090058 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.089998 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.090058 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.090023 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.090175 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.090068 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.090229 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.090208 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.090280 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.090235 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.090331 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.090275 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.090382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.090335 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.191737 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191645 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.191917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191749 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svh5s\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-kube-api-access-svh5s\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.191917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191794 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.191917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191814 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.191917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191846 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.191917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191889 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.191917 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191908 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.192256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191948 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.192256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.191984 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.192412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.192388 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.192597 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.192459 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.192656 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.192496 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.192711 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.192662 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.192759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.192684 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.194859 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.194838 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.194948 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.194926 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.265507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.265420 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svh5s\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-kube-api-access-svh5s\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.267648 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.267610 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.539791 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.539718 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:35.969309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.969280 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm"] Oct 31 00:56:35.973715 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:56:35.973664 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1aeb40_adca_499b_90ae_59bbb661fbf1.slice/crio-06ee58f6275997956175473637dd9909fbfd03edd69a194f84963659fda9cbcf WatchSource:0}: Error finding container 06ee58f6275997956175473637dd9909fbfd03edd69a194f84963659fda9cbcf: Status 404 returned error can't find the container with id 06ee58f6275997956175473637dd9909fbfd03edd69a194f84963659fda9cbcf Oct 31 00:56:35.976071 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.976043 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:56:35.976150 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.976112 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:56:35.976150 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:35.976141 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:56:36.056808 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:36.055030 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr"] Oct 31 00:56:36.060592 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:36.060569 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3f42f1-vpbbr"] Oct 31 00:56:36.608903 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:36.608868 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61898d97-49da-49bb-a4fc-f8bcf6cf2997" path="/var/lib/kubelet/pods/61898d97-49da-49bb-a4fc-f8bcf6cf2997/volumes" Oct 31 00:56:36.741951 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:36.741915 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" event={"ID":"8e1aeb40-adca-499b-90ae-59bbb661fbf1","Type":"ContainerStarted","Data":"730e7da24e24be31eb536365df537ee466a297dd60ad9cf090bc5f46878380ee"} Oct 31 00:56:36.742118 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:36.741974 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" event={"ID":"8e1aeb40-adca-499b-90ae-59bbb661fbf1","Type":"ContainerStarted","Data":"06ee58f6275997956175473637dd9909fbfd03edd69a194f84963659fda9cbcf"} Oct 31 00:56:36.859764 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:36.859623 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" podStartSLOduration=2.859610777 podStartE2EDuration="2.859610777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:36.859105068 +0000 UTC m=+1678.395247376" watchObservedRunningTime="2026-10-31 00:56:36.859610777 +0000 UTC m=+1678.395753081" Oct 31 00:56:37.540367 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:37.540334 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:37.541986 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:37.541956 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:56:37.542103 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:37.542014 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:56:38.540279 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:38.540244 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:56:38.540451 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:38.540301 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:56:39.540719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:39.540662 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:56:39.541115 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:39.540740 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:56:40.540417 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:40.540386 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:56:40.540608 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:40.540434 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:56:40.626718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:40.622624 5719 scope.go:122] "RemoveContainer" containerID="bdc1c22f7f55ac2e2fcc720b0614e339a57187d171fa56f21d8d96a1bf06b098" Oct 31 00:56:41.118718 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.118159 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j"] Oct 31 00:56:41.129159 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.129118 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.173170 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.173139 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.173170 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.173153 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j"] Oct 31 00:56:41.173371 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.173351 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd4qg\" (UniqueName: \"kubernetes.io/projected/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-kube-api-access-fd4qg\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.173443 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.173420 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.225834 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.225165 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:56:41.226130 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.226075 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:56:41.226940 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.226910 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:56:41.227163 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.227146 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:56:41.274671 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.274647 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd4qg\" (UniqueName: \"kubernetes.io/projected/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-kube-api-access-fd4qg\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.274782 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.274730 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.274873 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.274817 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.275467 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.275443 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.277271 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.277252 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.334328 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.334307 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd4qg\" (UniqueName: \"kubernetes.io/projected/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-kube-api-access-fd4qg\") pod \"maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.443218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.443150 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:41.543731 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.540988 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 31 00:56:41.543731 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.541031 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 31 00:56:41.845114 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:41.845089 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j"] Oct 31 00:56:41.847012 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:56:41.846982 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3ac5a5_1bf6_4f0e_ac17_f9aa72303eb2.slice/crio-ce159488b6718f7fc0d0f703600c3c486736dead31c9a3206d4a4571854cb5d5 WatchSource:0}: Error finding container ce159488b6718f7fc0d0f703600c3c486736dead31c9a3206d4a4571854cb5d5: Status 404 returned error can't find the container with id ce159488b6718f7fc0d0f703600c3c486736dead31c9a3206d4a4571854cb5d5 Oct 31 00:56:42.056909 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.056877 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm"] Oct 31 00:56:42.059312 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.059293 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a994e802-54jkm"] Oct 31 00:56:42.545581 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.545508 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:42.545777 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.545759 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:42.546299 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.546281 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:56:42.603915 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.603879 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fa54b6-9b16-4614-9d5a-08955c8821b6" path="/var/lib/kubelet/pods/01fa54b6-9b16-4614-9d5a-08955c8821b6/volumes" Oct 31 00:56:42.787180 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.787141 5719 generic.go:356] "Generic (PLEG): container finished" podID="2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2" containerID="4a12a7c6ace2e353c46c38a424f8562ae8ecd5ad6cb63201cc5759709656a6a8" exitCode=0 Oct 31 00:56:42.787813 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.787768 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" event={"ID":"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2","Type":"ContainerDied","Data":"4a12a7c6ace2e353c46c38a424f8562ae8ecd5ad6cb63201cc5759709656a6a8"} Oct 31 00:56:42.787961 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:42.787889 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" event={"ID":"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2","Type":"ContainerStarted","Data":"ce159488b6718f7fc0d0f703600c3c486736dead31c9a3206d4a4571854cb5d5"} Oct 31 00:56:43.941423 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:43.941396 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:44.017259 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.017220 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-xks-ca\") pod \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " Oct 31 00:56:44.017490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.017303 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-kube-api-access-fd4qg\" (UniqueName: \"kubernetes.io/projected/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-kube-api-access-fd4qg\") pod \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " Oct 31 00:56:44.017490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.017391 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-openshift-ca\") pod \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\" (UID: \"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2\") " Oct 31 00:56:44.018382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.018352 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-openshift-ca" pod "2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2" (UID: "2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:44.023262 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.023220 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-kube-api-access-fd4qg" pod "2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2" (UID: "2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2"). InnerVolumeSpecName "kube-api-access-fd4qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:44.023412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.023256 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-xks-ca" pod "2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2" (UID: "2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:44.118820 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.118789 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:44.118820 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.118815 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd4qg\" (UniqueName: \"kubernetes.io/projected/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-kube-api-access-fd4qg\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:44.118820 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.118828 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:44.802135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.802105 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" event={"ID":"2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2","Type":"ContainerDied","Data":"ce159488b6718f7fc0d0f703600c3c486736dead31c9a3206d4a4571854cb5d5"} Oct 31 00:56:44.802135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.802131 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j" Oct 31 00:56:44.802352 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:44.802159 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce159488b6718f7fc0d0f703600c3c486736dead31c9a3206d4a4571854cb5d5" Oct 31 00:56:45.511994 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:45.511957 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9"] Oct 31 00:56:45.559249 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:45.559207 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69a4bc-cvdq9"] Oct 31 00:56:46.609124 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:46.609080 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1741a680-9746-480e-b52e-e1d7ce3d4ae6" path="/var/lib/kubelet/pods/1741a680-9746-480e-b52e-e1d7ce3d4ae6/volumes" Oct 31 00:56:46.697079 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:46.697048 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v"] Oct 31 00:56:46.744512 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:46.744486 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3c636f-mlb9v"] Oct 31 00:56:48.602994 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:48.602963 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e" path="/var/lib/kubelet/pods/28ae4c9b-e9ee-4b59-b3bd-fa2e0634493e/volumes" Oct 31 00:56:50.038410 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.038383 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm"] Oct 31 00:56:50.039017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.039002 5719 cpu_manager.go:414] "containerMap: removing container" podUID="2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2" containerName="revoke-keys" Oct 31 00:56:50.039057 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.039019 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2" containerName="revoke-keys" Oct 31 00:56:50.039142 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.039133 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2" containerName="revoke-keys" Oct 31 00:56:50.043573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.043557 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.146105 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.146077 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm"] Oct 31 00:56:50.189309 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189280 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.189458 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189386 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.189458 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189425 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q56hg\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-kube-api-access-q56hg\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.189541 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189513 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.189617 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189586 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.189780 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189670 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.189780 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189762 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.189936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189833 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.189936 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.189897 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291191 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291115 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291191 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291162 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291203 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291221 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291285 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291336 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291383 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291377 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q56hg\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-kube-api-access-q56hg\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291659 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291417 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291659 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291478 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291852 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291813 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291852 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291827 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.291971 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.291955 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.292234 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.292211 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.292324 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.292243 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.294208 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.294187 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.294301 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.294214 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.350188 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.350162 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.350414 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.350396 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q56hg\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-kube-api-access-q56hg\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.356225 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.356209 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:50.781603 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.781570 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm"] Oct 31 00:56:50.786364 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:56:50.786342 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9f64a1_9705_47b6_a6c9_c9b69edfba15.slice/crio-07fc277fa5c91ec4e276d64ed0abffdb7c7ffda3b89228ceb4786a0e4834a172 WatchSource:0}: Error finding container 07fc277fa5c91ec4e276d64ed0abffdb7c7ffda3b89228ceb4786a0e4834a172: Status 404 returned error can't find the container with id 07fc277fa5c91ec4e276d64ed0abffdb7c7ffda3b89228ceb4786a0e4834a172 Oct 31 00:56:50.788755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.788716 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:56:50.788887 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.788797 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:56:50.788887 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.788841 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:56:50.838566 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:50.838542 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" event={"ID":"9d9f64a1-9705-47b6-a6c9-c9b69edfba15","Type":"ContainerStarted","Data":"07fc277fa5c91ec4e276d64ed0abffdb7c7ffda3b89228ceb4786a0e4834a172"} Oct 31 00:56:51.846006 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:51.845968 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" event={"ID":"9d9f64a1-9705-47b6-a6c9-c9b69edfba15","Type":"ContainerStarted","Data":"ea33bf3f958c7f79b3b4eaf7501827bdaea8bd46b90837734444474e100445af"} Oct 31 00:56:52.021719 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:52.018110 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" podStartSLOduration=3.018086755 podStartE2EDuration="3.018086755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:52.015282632 +0000 UTC m=+1693.551424938" watchObservedRunningTime="2026-10-31 00:56:52.018086755 +0000 UTC m=+1693.554229061" Oct 31 00:56:52.356767 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:52.356735 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:52.357679 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:52.357655 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:52.357831 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:52.357727 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 00:56:53.360021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:53.359072 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:53.360021 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:53.359132 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 00:56:54.359714 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.358096 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:54.359714 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.358190 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 00:56:54.560179 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.560147 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw"] Oct 31 00:56:54.564069 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.564052 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.665839 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.665748 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:56:54.666017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.665843 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:56:54.666017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.665854 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:56:54.666017 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.665968 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:56:54.668611 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.668589 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw"] Oct 31 00:56:54.670881 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.670852 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11771db5-0f69-4812-bab7-2db041827003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.671097 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.671076 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11771db5-0f69-4812-bab7-2db041827003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.671247 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.671231 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfzs\" (UniqueName: \"kubernetes.io/projected/11771db5-0f69-4812-bab7-2db041827003-kube-api-access-9qfzs\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.772442 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.772406 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11771db5-0f69-4812-bab7-2db041827003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.772881 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.772859 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11771db5-0f69-4812-bab7-2db041827003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.773143 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.773118 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfzs\" (UniqueName: \"kubernetes.io/projected/11771db5-0f69-4812-bab7-2db041827003-kube-api-access-9qfzs\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.773258 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.773232 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11771db5-0f69-4812-bab7-2db041827003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.775290 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.775270 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11771db5-0f69-4812-bab7-2db041827003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.832054 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.832033 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfzs\" (UniqueName: \"kubernetes.io/projected/11771db5-0f69-4812-bab7-2db041827003-kube-api-access-9qfzs\") pod \"maas-api-revoke-keys-e2e-api-b-e942ab-5njqw\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:54.878986 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:54.878960 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:55.417067 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:55.412012 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:55.417067 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:55.412884 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 00:56:55.545120 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:55.545083 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw"] Oct 31 00:56:55.896715 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:55.895739 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" event={"ID":"11771db5-0f69-4812-bab7-2db041827003","Type":"ContainerStarted","Data":"007af320a9d97165cb5929ee5ca567c5399f9cc1d2856a2f3f6a14fec8b36664"} Oct 31 00:56:56.357378 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:56.357340 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 00:56:56.357564 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:56.357404 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 00:56:56.906413 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:56.906374 5719 generic.go:356] "Generic (PLEG): container finished" podID="11771db5-0f69-4812-bab7-2db041827003" containerID="51afcb1100164c5a3df5de21251eefd3ecfa5e2c06fa0677bfa16d9e9d4ac7f0" exitCode=0 Oct 31 00:56:56.906933 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:56.906462 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" event={"ID":"11771db5-0f69-4812-bab7-2db041827003","Type":"ContainerDied","Data":"51afcb1100164c5a3df5de21251eefd3ecfa5e2c06fa0677bfa16d9e9d4ac7f0"} Oct 31 00:56:57.552013 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:57.551982 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:57.552456 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:57.552434 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:57.553398 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:57.553382 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:56:58.114366 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.114342 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:58.183174 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.183096 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11771db5-0f69-4812-bab7-2db041827003-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11771db5-0f69-4812-bab7-2db041827003-maas-api-xks-ca\") pod \"11771db5-0f69-4812-bab7-2db041827003\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " Oct 31 00:56:58.183322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.183263 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11771db5-0f69-4812-bab7-2db041827003-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11771db5-0f69-4812-bab7-2db041827003-maas-api-openshift-ca\") pod \"11771db5-0f69-4812-bab7-2db041827003\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " Oct 31 00:56:58.183322 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.183291 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11771db5-0f69-4812-bab7-2db041827003-kube-api-access-9qfzs\" (UniqueName: \"kubernetes.io/projected/11771db5-0f69-4812-bab7-2db041827003-kube-api-access-9qfzs\") pod \"11771db5-0f69-4812-bab7-2db041827003\" (UID: \"11771db5-0f69-4812-bab7-2db041827003\") " Oct 31 00:56:58.183639 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.183616 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11771db5-0f69-4812-bab7-2db041827003-maas-api-openshift-ca" pod "11771db5-0f69-4812-bab7-2db041827003" (UID: "11771db5-0f69-4812-bab7-2db041827003"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:56:58.183879 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.183844 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11771db5-0f69-4812-bab7-2db041827003-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.185385 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.185359 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11771db5-0f69-4812-bab7-2db041827003-maas-api-xks-ca" pod "11771db5-0f69-4812-bab7-2db041827003" (UID: "11771db5-0f69-4812-bab7-2db041827003"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:56:58.185538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.185519 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11771db5-0f69-4812-bab7-2db041827003-kube-api-access-9qfzs" pod "11771db5-0f69-4812-bab7-2db041827003" (UID: "11771db5-0f69-4812-bab7-2db041827003"). InnerVolumeSpecName "kube-api-access-9qfzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:56:58.285296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.285267 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11771db5-0f69-4812-bab7-2db041827003-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.285296 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.285295 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfzs\" (UniqueName: \"kubernetes.io/projected/11771db5-0f69-4812-bab7-2db041827003-kube-api-access-9qfzs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:56:58.922085 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.922011 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" Oct 31 00:56:58.922085 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.922046 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw" event={"ID":"11771db5-0f69-4812-bab7-2db041827003","Type":"ContainerDied","Data":"007af320a9d97165cb5929ee5ca567c5399f9cc1d2856a2f3f6a14fec8b36664"} Oct 31 00:56:58.922372 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:56:58.922102 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="007af320a9d97165cb5929ee5ca567c5399f9cc1d2856a2f3f6a14fec8b36664" Oct 31 00:57:03.011811 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:57:03.011776 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:57:05.170851 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.170821 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx"] Oct 31 00:57:05.171387 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.171371 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11771db5-0f69-4812-bab7-2db041827003" containerName="revoke-keys" Oct 31 00:57:05.171434 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.171389 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11771db5-0f69-4812-bab7-2db041827003" containerName="revoke-keys" Oct 31 00:57:05.171490 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.171480 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11771db5-0f69-4812-bab7-2db041827003" containerName="revoke-keys" Oct 31 00:57:05.174959 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.174941 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.233659 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.233631 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx"] Oct 31 00:57:05.318289 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318260 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.318434 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318310 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpcn6\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-kube-api-access-jpcn6\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.318485 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318461 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e117cdd-ec64-48e1-a808-fec433672cfe-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.318530 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318496 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.318575 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318561 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.318632 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318618 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.318758 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318735 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.318815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318766 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.318815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.318792 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420387 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420415 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420447 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420479 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420541 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpcn6\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-kube-api-access-jpcn6\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420840 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420675 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e117cdd-ec64-48e1-a808-fec433672cfe-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420840 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420730 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420840 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420783 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.420840 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420813 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.421072 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.420840 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.421420 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.421366 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e117cdd-ec64-48e1-a808-fec433672cfe-istiod-ca-cert\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.421561 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.421469 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-certs\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.421561 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.421435 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-data\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.421671 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.421628 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-credential-socket\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.423165 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.423146 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-envoy\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.423248 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.423170 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-podinfo\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.479411 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.479391 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-token\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.479527 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.479504 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpcn6\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-kube-api-access-jpcn6\") pod \"e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.488617 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.488599 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:05.917750 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.917718 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx"] Oct 31 00:57:05.958469 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:57:05.958431 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e117cdd_ec64_48e1_a808_fec433672cfe.slice/crio-b427a400e74b10cfc21db136ebc3fcf0b773e569fe788673fca8766632021580 WatchSource:0}: Error finding container b427a400e74b10cfc21db136ebc3fcf0b773e569fe788673fca8766632021580: Status 404 returned error can't find the container with id b427a400e74b10cfc21db136ebc3fcf0b773e569fe788673fca8766632021580 Oct 31 00:57:05.961271 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.961234 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:57:05.961365 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.961322 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:57:05.961404 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:05.961364 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:57:06.030529 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:06.030488 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" event={"ID":"7e117cdd-ec64-48e1-a808-fec433672cfe","Type":"ContainerStarted","Data":"c03df18638808a5617ba471d82c34bd8caf8ba30c8344912a28b0b5aeaed24b0"} Oct 31 00:57:06.030641 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:06.030557 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" event={"ID":"7e117cdd-ec64-48e1-a808-fec433672cfe","Type":"ContainerStarted","Data":"b427a400e74b10cfc21db136ebc3fcf0b773e569fe788673fca8766632021580"} Oct 31 00:57:06.148573 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:06.148516 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podStartSLOduration=1.148501376 podStartE2EDuration="1.148501376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:06.146961035 +0000 UTC m=+1707.683103342" watchObservedRunningTime="2026-10-31 00:57:06.148501376 +0000 UTC m=+1707.684643746" Oct 31 00:57:06.488891 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:06.488855 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:07.490612 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:07.490572 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:57:07.490999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:07.490641 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:57:08.489289 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:08.489256 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:57:08.489488 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:08.489308 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:57:09.488995 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:09.488967 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:57:09.489394 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:09.489015 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:57:10.047776 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:10.047739 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:57:10.489064 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:10.489029 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:57:10.489479 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:10.489091 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:57:11.489290 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:11.489251 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:57:11.489722 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:11.489319 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:57:12.489168 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:12.489123 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:57:12.489439 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:12.489409 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Oct 31 00:57:13.507887 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:13.507853 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:13.508978 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:13.508952 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:13.510287 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:13.510265 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:57:18.048262 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:18.048225 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj"] Oct 31 00:57:18.095669 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:18.095645 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1bb633d-qqqmj"] Oct 31 00:57:18.602793 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:18.602761 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8984cf1-62df-4938-a90a-2a129e9b5e2d" path="/var/lib/kubelet/pods/d8984cf1-62df-4938-a90a-2a129e9b5e2d/volumes" Oct 31 00:57:22.341841 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:22.341798 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j"] Oct 31 00:57:22.342357 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:22.341862 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7e6fe88b-kv97j"] Oct 31 00:57:22.603412 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:22.603347 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2" path="/var/lib/kubelet/pods/2c3ac5a5-1bf6-4f0e-ac17-f9aa72303eb2/volumes" Oct 31 00:57:28.048235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:28.048202 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66"] Oct 31 00:57:28.096369 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:28.096328 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d840f0f-gzh66"] Oct 31 00:57:28.605209 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:28.605179 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51c0e76-2466-4ed7-951f-338440df990f" path="/var/lib/kubelet/pods/f51c0e76-2466-4ed7-951f-338440df990f/volumes" Oct 31 00:57:33.010070 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:57:33.009995 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:57:40.659850 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:40.659817 5719 scope.go:122] "RemoveContainer" containerID="d3cd33e1f0710fce84e7cce5dae8c9078219993f24c678688aa0f24c4c01f5b1" Oct 31 00:57:40.669248 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:40.669228 5719 scope.go:122] "RemoveContainer" containerID="3cb65d1bd743bdc0d702e072a9f4ac773816cec4c83cb3f092c1753a9fe1590f" Oct 31 00:57:40.678059 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:57:40.678038 5719 scope.go:122] "RemoveContainer" containerID="e6b11912d29d4cfa82f2b21bbe0b8b88044b6c7f5324448ae23d2b7e20023cde" Oct 31 00:58:03.011635 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:58:03.011595 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:58:09.848665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:09.848625 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:58:10.050682 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:10.050650 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh"] Oct 31 00:58:10.053045 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:10.053022 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-780dd372-7h8sh"] Oct 31 00:58:10.603836 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:10.603805 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="104eb118-8eaf-4aa3-8036-8d61ead2e690" path="/var/lib/kubelet/pods/104eb118-8eaf-4aa3-8036-8d61ead2e690/volumes" Oct 31 00:58:12.233591 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.233557 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm"] Oct 31 00:58:12.234067 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.233797 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" containerID="cri-o://730e7da24e24be31eb536365df537ee466a297dd60ad9cf090bc5f46878380ee" gracePeriod=30 Oct 31 00:58:12.247991 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.247964 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm"] Oct 31 00:58:12.248184 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.248165 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" containerID="cri-o://ea33bf3f958c7f79b3b4eaf7501827bdaea8bd46b90837734444474e100445af" gracePeriod=30 Oct 31 00:58:12.263362 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.263342 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx"] Oct 31 00:58:12.263530 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.263511 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" containerID="cri-o://c03df18638808a5617ba471d82c34bd8caf8ba30c8344912a28b0b5aeaed24b0" gracePeriod=30 Oct 31 00:58:12.548510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.548434 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:58:12.548510 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.548485 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:58:12.554582 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.554552 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:58:12.554739 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:12.554614 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:58:13.511996 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:13.511963 5719 patch_prober.go:29] interesting pod/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:58:13.512382 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:13.512017 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:58:14.977469 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:14.977436 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk"] Oct 31 00:58:14.977912 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:14.977637 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" containerID="cri-o://823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53" gracePeriod=30 Oct 31 00:58:17.514616 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.514573 5719 generic.go:356] "Generic (PLEG): container finished" podID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerID="ea33bf3f958c7f79b3b4eaf7501827bdaea8bd46b90837734444474e100445af" exitCode=0 Oct 31 00:58:17.516565 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.514674 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" event={"ID":"9d9f64a1-9705-47b6-a6c9-c9b69edfba15","Type":"ContainerDied","Data":"ea33bf3f958c7f79b3b4eaf7501827bdaea8bd46b90837734444474e100445af"} Oct 31 00:58:17.518665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.518408 5719 generic.go:356] "Generic (PLEG): container finished" podID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerID="c03df18638808a5617ba471d82c34bd8caf8ba30c8344912a28b0b5aeaed24b0" exitCode=0 Oct 31 00:58:17.518665 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.518597 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" event={"ID":"7e117cdd-ec64-48e1-a808-fec433672cfe","Type":"ContainerDied","Data":"c03df18638808a5617ba471d82c34bd8caf8ba30c8344912a28b0b5aeaed24b0"} Oct 31 00:58:17.523592 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.523428 5719 generic.go:356] "Generic (PLEG): container finished" podID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerID="730e7da24e24be31eb536365df537ee466a297dd60ad9cf090bc5f46878380ee" exitCode=0 Oct 31 00:58:17.523592 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.523530 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" event={"ID":"8e1aeb40-adca-499b-90ae-59bbb661fbf1","Type":"ContainerDied","Data":"730e7da24e24be31eb536365df537ee466a297dd60ad9cf090bc5f46878380ee"} Oct 31 00:58:17.564256 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.564163 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:58:17.571999 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.571976 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:58:17.605866 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.605849 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:58:17.712927 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.712897 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-kube-api-access-svh5s\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-kube-api-access-svh5s\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.713049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.712949 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e117cdd-ec64-48e1-a808-fec433672cfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e117cdd-ec64-48e1-a808-fec433672cfe-istiod-ca-cert\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.713049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.712981 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-socket\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.713049 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713019 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-certs\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.713219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713047 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-credential-socket\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.713219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713077 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-podinfo\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.713219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713148 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-certs\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.713219 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713192 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-token\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-token\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.713427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713227 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istiod-ca-cert\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.713427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713259 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-kube-api-access-q56hg\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-kube-api-access-q56hg\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.713427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713287 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-kube-api-access-jpcn6\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-kube-api-access-jpcn6\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.713427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713321 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-certs" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.713427 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713330 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-credential-socket\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713433 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-data\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713442 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e117cdd-ec64-48e1-a808-fec433672cfe-istiod-ca-cert" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713473 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istiod-ca-cert\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713512 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-credential-socket\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713532 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-credential-socket" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713546 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-token\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-token\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713583 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istiod-ca-cert" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713605 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-data\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.713673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713649 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-podinfo\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.714148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713736 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-data\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.714148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713813 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-envoy\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.714148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713855 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-socket" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.714148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713873 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-envoy\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.714148 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.713893 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-certs" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.714463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.714227 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istiod-ca-cert" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:17.714463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.714441 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-data" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.714463 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.714484 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-certs\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.714759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.714629 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-podinfo\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.714759 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.714722 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-envoy\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.714874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.714761 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-socket\") pod \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\" (UID: \"8e1aeb40-adca-499b-90ae-59bbb661fbf1\") " Oct 31 00:58:17.714874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.714790 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-token\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-token\") pod \"7e117cdd-ec64-48e1-a808-fec433672cfe\" (UID: \"7e117cdd-ec64-48e1-a808-fec433672cfe\") " Oct 31 00:58:17.714874 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.714862 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-socket\") pod \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\" (UID: \"9d9f64a1-9705-47b6-a6c9-c9b69edfba15\") " Oct 31 00:58:17.715360 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715099 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-credential-socket" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715731 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e117cdd-ec64-48e1-a808-fec433672cfe-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715754 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715770 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715785 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715801 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715815 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715830 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715845 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716106 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.715861 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.716742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.716176 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-kube-api-access-svh5s" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "kube-api-access-svh5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.716742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.716434 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-token" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.716742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.716553 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-envoy" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.716742 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.716585 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-certs" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.716979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.716752 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-credential-socket" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.716979 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.716816 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-data" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.717931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.717218 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-token" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.717931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.717246 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-data" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.717931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.717645 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-kube-api-access-q56hg" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "kube-api-access-q56hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.717931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.717650 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-socket" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.717931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.717806 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-podinfo" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:58:17.717931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.717900 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-socket" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.717931 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.717910 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-kube-api-access-jpcn6" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "kube-api-access-jpcn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.718954 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.718930 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-podinfo" pod "8e1aeb40-adca-499b-90ae-59bbb661fbf1" (UID: "8e1aeb40-adca-499b-90ae-59bbb661fbf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:58:17.719131 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.719113 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-podinfo" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:58:17.719552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.719537 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-envoy" pod "9d9f64a1-9705-47b6-a6c9-c9b69edfba15" (UID: "9d9f64a1-9705-47b6-a6c9-c9b69edfba15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.720149 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.720132 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-envoy" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:17.720212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.720150 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-token" pod "7e117cdd-ec64-48e1-a808-fec433672cfe" (UID: "7e117cdd-ec64-48e1-a808-fec433672cfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:17.817076 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817051 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817077 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svh5s\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-kube-api-access-svh5s\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817088 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817096 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817106 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q56hg\" (UniqueName: \"kubernetes.io/projected/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-kube-api-access-q56hg\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817115 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpcn6\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-kube-api-access-jpcn6\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817123 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817133 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817141 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817149 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817157 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817165 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817172 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817180 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817188 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d9f64a1-9705-47b6-a6c9-c9b69edfba15-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817196 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817205 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e1aeb40-adca-499b-90ae-59bbb661fbf1-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:17.817213 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:17.817212 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e117cdd-ec64-48e1-a808-fec433672cfe-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:18.531943 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.531869 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" Oct 31 00:58:18.531943 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.531874 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm" event={"ID":"9d9f64a1-9705-47b6-a6c9-c9b69edfba15","Type":"ContainerDied","Data":"07fc277fa5c91ec4e276d64ed0abffdb7c7ffda3b89228ceb4786a0e4834a172"} Oct 31 00:58:18.531943 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.531937 5719 scope.go:122] "RemoveContainer" containerID="ea33bf3f958c7f79b3b4eaf7501827bdaea8bd46b90837734444474e100445af" Oct 31 00:58:18.533396 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.533377 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" Oct 31 00:58:18.533505 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.533417 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx" event={"ID":"7e117cdd-ec64-48e1-a808-fec433672cfe","Type":"ContainerDied","Data":"b427a400e74b10cfc21db136ebc3fcf0b773e569fe788673fca8766632021580"} Oct 31 00:58:18.541073 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.541050 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" event={"ID":"8e1aeb40-adca-499b-90ae-59bbb661fbf1","Type":"ContainerDied","Data":"06ee58f6275997956175473637dd9909fbfd03edd69a194f84963659fda9cbcf"} Oct 31 00:58:18.541176 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.541091 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm" Oct 31 00:58:18.543030 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.542854 5719 scope.go:122] "RemoveContainer" containerID="c03df18638808a5617ba471d82c34bd8caf8ba30c8344912a28b0b5aeaed24b0" Oct 31 00:58:18.552526 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.552511 5719 scope.go:122] "RemoveContainer" containerID="730e7da24e24be31eb536365df537ee466a297dd60ad9cf090bc5f46878380ee" Oct 31 00:58:18.747926 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.747896 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm"] Oct 31 00:58:18.750673 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.750650 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-ctmwm"] Oct 31 00:58:18.862538 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.862504 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx"] Oct 31 00:58:18.868662 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.868639 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-q82cx"] Oct 31 00:58:18.977489 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.977460 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm"] Oct 31 00:58:18.984024 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:18.983994 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7e6fe88b-openshift-default-5b64b75c89-6jphm"] Oct 31 00:58:20.242868 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.242814 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:58:20.344755 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.344731 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-token\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-token\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.344886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.344770 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-podinfo\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.344886 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.344826 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-data\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.345001 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.344985 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-socket\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.345059 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345024 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istiod-ca-cert\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.345105 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345056 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-envoy\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.345156 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345100 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-certs\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.345156 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345119 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-data" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:20.345507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345269 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-socket" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:20.345507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345352 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-certs" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:20.345507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345400 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istiod-ca-cert" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:20.345507 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345182 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-kube-api-access-4bsh8\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-kube-api-access-4bsh8\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.345844 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.345638 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-credential-socket\") pod \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\" (UID: \"7f5ebf5e-00cf-4b8c-88d0-639967671f62\") " Oct 31 00:58:20.346135 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.346109 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-credential-socket" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:20.346350 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.346329 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.346422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.346356 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.346422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.346372 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.346422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.346387 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.346422 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.346402 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.347117 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.347093 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-podinfo" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:58:20.347235 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.347221 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-token" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:20.347291 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.347253 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-envoy" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:20.347574 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.347560 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-kube-api-access-4bsh8" pod "7f5ebf5e-00cf-4b8c-88d0-639967671f62" (UID: "7f5ebf5e-00cf-4b8c-88d0-639967671f62"). InnerVolumeSpecName "kube-api-access-4bsh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:20.448031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.447996 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.448031 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.448035 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bsh8\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-kube-api-access-4bsh8\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.448205 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.448051 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.448205 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.448069 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f5ebf5e-00cf-4b8c-88d0-639967671f62-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:58:20.566704 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.566646 5719 generic.go:356] "Generic (PLEG): container finished" podID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerID="823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53" exitCode=0 Oct 31 00:58:20.566876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.566720 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" Oct 31 00:58:20.566876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.566730 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" event={"ID":"7f5ebf5e-00cf-4b8c-88d0-639967671f62","Type":"ContainerDied","Data":"823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53"} Oct 31 00:58:20.566876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.566814 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk" event={"ID":"7f5ebf5e-00cf-4b8c-88d0-639967671f62","Type":"ContainerDied","Data":"09545a350480198abcc2fe2e8db24783735d9145276954387b59234d78517136"} Oct 31 00:58:20.566876 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.566837 5719 scope.go:122] "RemoveContainer" containerID="823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53" Oct 31 00:58:20.578261 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.578244 5719 scope.go:122] "RemoveContainer" containerID="823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53" Oct 31 00:58:20.578530 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:58:20.578513 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53\": container with ID starting with 823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53 not found: ID does not exist" containerID="823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53" Oct 31 00:58:20.578581 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.578540 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53"} err="failed to get container status \"823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53\": rpc error: code = NotFound desc = could not find container \"823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53\": container with ID starting with 823ebc4bdfdf1c6d61bf398ff29c03f8524a03dec661c502f925183098a2ae53 not found: ID does not exist" Oct 31 00:58:20.603961 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.603936 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" path="/var/lib/kubelet/pods/7e117cdd-ec64-48e1-a808-fec433672cfe/volumes" Oct 31 00:58:20.604429 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.604414 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" path="/var/lib/kubelet/pods/8e1aeb40-adca-499b-90ae-59bbb661fbf1/volumes" Oct 31 00:58:20.604873 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.604859 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" path="/var/lib/kubelet/pods/9d9f64a1-9705-47b6-a6c9-c9b69edfba15/volumes" Oct 31 00:58:20.724714 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.724672 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk"] Oct 31 00:58:20.728746 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:20.728712 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-w8cqk"] Oct 31 00:58:22.604186 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:22.604152 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" path="/var/lib/kubelet/pods/7f5ebf5e-00cf-4b8c-88d0-639967671f62/volumes" Oct 31 00:58:33.009432 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:58:33.009394 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:58:40.710043 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:40.710014 5719 scope.go:122] "RemoveContainer" containerID="c79d1ee2f41e4b2cd20b69022d8924160327c945308cd043fb66ba7783fc8190" Oct 31 00:58:40.719211 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:40.719193 5719 scope.go:122] "RemoveContainer" containerID="e226c781ea7fe30adf33f79b6ccd30e90ef093f43f02295fd4a931c9fd1f209d" Oct 31 00:58:53.049314 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:53.049230 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j"] Oct 31 00:58:53.096015 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:53.095985 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11c4c29-bt56j"] Oct 31 00:58:54.413794 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:54.413765 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw"] Oct 31 00:58:54.461395 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:54.461371 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e942ab-5njqw"] Oct 31 00:58:54.601801 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:54.601775 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11771db5-0f69-4812-bab7-2db041827003" path="/var/lib/kubelet/pods/11771db5-0f69-4812-bab7-2db041827003/volumes" Oct 31 00:58:54.602216 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:58:54.602202 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89bceddb-a966-41d9-a6ec-13922f37d7c4" path="/var/lib/kubelet/pods/89bceddb-a966-41d9-a6ec-13922f37d7c4/volumes" Oct 31 00:59:03.010457 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:59:03.010419 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:59:06.177728 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.177669 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx"] Oct 31 00:59:06.178373 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178354 5719 cpu_manager.go:414] "containerMap: removing container" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" Oct 31 00:59:06.178433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178377 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" Oct 31 00:59:06.178433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178401 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" Oct 31 00:59:06.178433 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178407 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" Oct 31 00:59:06.178532 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178440 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" Oct 31 00:59:06.178532 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178448 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" Oct 31 00:59:06.178532 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178475 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" Oct 31 00:59:06.178532 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178480 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" Oct 31 00:59:06.178649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178573 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9f64a1-9705-47b6-a6c9-c9b69edfba15" containerName="istio-proxy" Oct 31 00:59:06.178649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178582 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e117cdd-ec64-48e1-a808-fec433672cfe" containerName="istio-proxy" Oct 31 00:59:06.178649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178589 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1aeb40-adca-499b-90ae-59bbb661fbf1" containerName="istio-proxy" Oct 31 00:59:06.178649 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.178601 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5ebf5e-00cf-4b8c-88d0-639967671f62" containerName="istio-proxy" Oct 31 00:59:06.181904 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.181884 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.236717 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.236675 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:59:06.276552 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.276520 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.276679 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.276566 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.276679 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.276641 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrglg\" (UniqueName: \"kubernetes.io/projected/e543c58e-dad9-4c1d-af9b-27aa5849c998-kube-api-access-jrglg\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.284630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.284596 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:59:06.284630 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.284625 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:06.284854 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.284675 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:59:06.286980 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.286962 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx"] Oct 31 00:59:06.377655 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.377629 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.377788 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.377668 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.377788 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.377735 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrglg\" (UniqueName: \"kubernetes.io/projected/e543c58e-dad9-4c1d-af9b-27aa5849c998-kube-api-access-jrglg\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.378269 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.378247 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.380212 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.380178 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.441243 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.441159 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrglg\" (UniqueName: \"kubernetes.io/projected/e543c58e-dad9-4c1d-af9b-27aa5849c998-kube-api-access-jrglg\") pod \"maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.495415 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.495380 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:06.879264 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.879229 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx"] Oct 31 00:59:06.885136 ip-10-0-129-8 kubenswrapper[5719]: W1031 00:59:06.885104 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode543c58e_dad9_4c1d_af9b_27aa5849c998.slice/crio-57b2d46df51b00d7f80aa89a36318bbed156c1d15b4a9f64a0f70176851e5c49 WatchSource:0}: Error finding container 57b2d46df51b00d7f80aa89a36318bbed156c1d15b4a9f64a0f70176851e5c49: Status 404 returned error can't find the container with id 57b2d46df51b00d7f80aa89a36318bbed156c1d15b4a9f64a0f70176851e5c49 Oct 31 00:59:06.895581 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:06.895558 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" event={"ID":"e543c58e-dad9-4c1d-af9b-27aa5849c998","Type":"ContainerStarted","Data":"57b2d46df51b00d7f80aa89a36318bbed156c1d15b4a9f64a0f70176851e5c49"} Oct 31 00:59:07.905321 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:07.905278 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" event={"ID":"e543c58e-dad9-4c1d-af9b-27aa5849c998","Type":"ContainerStarted","Data":"89582cd562374a6b0713d7cedf7a0423d4826b377b7a838529238fd458e63688"} Oct 31 00:59:08.019332 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:08.019278 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" podStartSLOduration=2.019264394 podStartE2EDuration="2.019264394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:59:08.017278471 +0000 UTC m=+1829.553420768" watchObservedRunningTime="2026-10-31 00:59:08.019264394 +0000 UTC m=+1829.555406698" Oct 31 00:59:08.913339 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:08.913310 5719 generic.go:356] "Generic (PLEG): container finished" podID="e543c58e-dad9-4c1d-af9b-27aa5849c998" containerID="89582cd562374a6b0713d7cedf7a0423d4826b377b7a838529238fd458e63688" exitCode=0 Oct 31 00:59:08.913738 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:08.913398 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" event={"ID":"e543c58e-dad9-4c1d-af9b-27aa5849c998","Type":"ContainerDied","Data":"89582cd562374a6b0713d7cedf7a0423d4826b377b7a838529238fd458e63688"} Oct 31 00:59:10.069299 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.069276 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:10.126958 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.126922 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e543c58e-dad9-4c1d-af9b-27aa5849c998-kube-api-access-jrglg\" (UniqueName: \"kubernetes.io/projected/e543c58e-dad9-4c1d-af9b-27aa5849c998-kube-api-access-jrglg\") pod \"e543c58e-dad9-4c1d-af9b-27aa5849c998\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " Oct 31 00:59:10.127123 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.127022 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-xks-ca\") pod \"e543c58e-dad9-4c1d-af9b-27aa5849c998\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " Oct 31 00:59:10.127123 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.127076 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-openshift-ca\") pod \"e543c58e-dad9-4c1d-af9b-27aa5849c998\" (UID: \"e543c58e-dad9-4c1d-af9b-27aa5849c998\") " Oct 31 00:59:10.127459 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.127432 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-openshift-ca" pod "e543c58e-dad9-4c1d-af9b-27aa5849c998" (UID: "e543c58e-dad9-4c1d-af9b-27aa5849c998"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:10.129191 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.129167 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-xks-ca" pod "e543c58e-dad9-4c1d-af9b-27aa5849c998" (UID: "e543c58e-dad9-4c1d-af9b-27aa5849c998"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:10.129289 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.129226 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e543c58e-dad9-4c1d-af9b-27aa5849c998-kube-api-access-jrglg" pod "e543c58e-dad9-4c1d-af9b-27aa5849c998" (UID: "e543c58e-dad9-4c1d-af9b-27aa5849c998"). InnerVolumeSpecName "kube-api-access-jrglg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:10.229218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.229134 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:10.229218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.229163 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrglg\" (UniqueName: \"kubernetes.io/projected/e543c58e-dad9-4c1d-af9b-27aa5849c998-kube-api-access-jrglg\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:10.229218 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.229174 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e543c58e-dad9-4c1d-af9b-27aa5849c998-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 00:59:10.926454 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.926382 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" Oct 31 00:59:10.926454 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.926408 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx" event={"ID":"e543c58e-dad9-4c1d-af9b-27aa5849c998","Type":"ContainerDied","Data":"57b2d46df51b00d7f80aa89a36318bbed156c1d15b4a9f64a0f70176851e5c49"} Oct 31 00:59:10.926454 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:10.926455 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b2d46df51b00d7f80aa89a36318bbed156c1d15b4a9f64a0f70176851e5c49" Oct 31 00:59:13.686335 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:13.686302 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 00:59:33.009980 ip-10-0-129-8 kubenswrapper[5719]: E1031 00:59:33.009916 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 00:59:40.770011 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:40.769976 5719 scope.go:122] "RemoveContainer" containerID="adc33bc01baa5ec3005b8934432720dc161016d5bb5342e429d2266962e33718" Oct 31 00:59:40.779154 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:40.779127 5719 scope.go:122] "RemoveContainer" containerID="6db4166722fa0ea3a36684ce59ef6aa87a24d14b3f432cd61dec4c137ccd9101" Oct 31 00:59:40.787358 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:40.787332 5719 scope.go:122] "RemoveContainer" containerID="e8ca1b26f1506f4d01caa5073d98c2d36d76c2ce7d4091b2ded4151e6e31f89d" Oct 31 00:59:50.847268 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:50.847237 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx"] Oct 31 00:59:50.894142 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:50.894107 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f66180-ttljx"] Oct 31 00:59:52.602791 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:52.602755 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e543c58e-dad9-4c1d-af9b-27aa5849c998" path="/var/lib/kubelet/pods/e543c58e-dad9-4c1d-af9b-27aa5849c998/volumes" Oct 31 00:59:59.440756 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.440727 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd"] Oct 31 00:59:59.441547 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.441527 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e543c58e-dad9-4c1d-af9b-27aa5849c998" containerName="revoke-keys" Oct 31 00:59:59.441596 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.441552 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e543c58e-dad9-4c1d-af9b-27aa5849c998" containerName="revoke-keys" Oct 31 00:59:59.441737 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.441724 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e543c58e-dad9-4c1d-af9b-27aa5849c998" containerName="revoke-keys" Oct 31 00:59:59.444615 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.444598 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.494677 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.494655 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:59:59.494815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.494765 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:59:59.494815 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.494786 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:59:59.521986 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.521961 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4mq\" (UniqueName: \"kubernetes.io/projected/ba77762d-91cf-43ba-89e2-e60ce150b366-kube-api-access-fz4mq\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.522093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.521998 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.522093 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.522076 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.543462 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.543265 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 00:59:59.567973 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.567939 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd"] Oct 31 00:59:59.623542 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.623514 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4mq\" (UniqueName: \"kubernetes.io/projected/ba77762d-91cf-43ba-89e2-e60ce150b366-kube-api-access-fz4mq\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.623749 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.623559 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.623749 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.623589 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.624343 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.624321 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.626008 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.625991 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.682781 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.682756 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4mq\" (UniqueName: \"kubernetes.io/projected/ba77762d-91cf-43ba-89e2-e60ce150b366-kube-api-access-fz4mq\") pod \"maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.756938 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.756858 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 00:59:59.992230 ip-10-0-129-8 kubenswrapper[5719]: I1031 00:59:59.992192 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:00:00.140408 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.140370 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd"] Oct 31 01:00:00.146274 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:00.146247 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba77762d_91cf_43ba_89e2_e60ce150b366.slice/crio-ecd829a0b5564dea119161571d1a21d5acc7f2c4d61ce814f08a25a7b41414bf WatchSource:0}: Error finding container ecd829a0b5564dea119161571d1a21d5acc7f2c4d61ce814f08a25a7b41414bf: Status 404 returned error can't find the container with id ecd829a0b5564dea119161571d1a21d5acc7f2c4d61ce814f08a25a7b41414bf Oct 31 01:00:00.163491 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.163447 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479"] Oct 31 01:00:00.168413 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.168390 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:00.168534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.168448 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg"] Oct 31 01:00:00.171419 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.171401 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:00.171516 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.171434 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd"] Oct 31 01:00:00.174192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.174177 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf"] Oct 31 01:00:00.174348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.174324 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:00.176982 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.176964 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:00.192606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.192585 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj"] Oct 31 01:00:00.195482 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.195466 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826"] Oct 31 01:00:00.195613 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.195595 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:00.198113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.198097 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:00.217226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.217204 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9"] Oct 31 01:00:00.220890 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.220859 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:00.221625 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.221605 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479"] Oct 31 01:00:00.231723 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.230350 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg"] Oct 31 01:00:00.241869 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.241845 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" event={"ID":"ba77762d-91cf-43ba-89e2-e60ce150b366","Type":"ContainerStarted","Data":"ecd829a0b5564dea119161571d1a21d5acc7f2c4d61ce814f08a25a7b41414bf"} Oct 31 01:00:00.247214 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.247174 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd"] Oct 31 01:00:00.274504 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.274471 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf"] Oct 31 01:00:00.282375 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.282346 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj"] Oct 31 01:00:00.287169 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.287070 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826"] Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333069 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf2jh\" (UniqueName: \"kubernetes.io/projected/8793d420-76d7-4900-ba05-fea167698c44-kube-api-access-zf2jh\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf\" (UID: \"8793d420-76d7-4900-ba05-fea167698c44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333150 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947986e0-5d7a-452c-b0ea-39d316fb6849-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826\" (UID: \"947986e0-5d7a-452c-b0ea-39d316fb6849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333187 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g767t\" (UniqueName: \"kubernetes.io/projected/947986e0-5d7a-452c-b0ea-39d316fb6849-kube-api-access-g767t\") pod \"maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826\" (UID: \"947986e0-5d7a-452c-b0ea-39d316fb6849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333226 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdlz\" (UniqueName: \"kubernetes.io/projected/8607532c-3977-469b-a040-7cd14ec8fefa-kube-api-access-spdlz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg\" (UID: \"8607532c-3977-469b-a040-7cd14ec8fefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333388 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjx2p\" (UniqueName: \"kubernetes.io/projected/7a54b41d-d663-41cd-9280-1b67dee25ce0-kube-api-access-gjx2p\") pod \"maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479\" (UID: \"7a54b41d-d663-41cd-9280-1b67dee25ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333511 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282243b1-7062-446f-a987-81624327f775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj\" (UID: \"282243b1-7062-446f-a987-81624327f775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333537 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8607532c-3977-469b-a040-7cd14ec8fefa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg\" (UID: \"8607532c-3977-469b-a040-7cd14ec8fefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333604 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh5cv\" (UniqueName: \"kubernetes.io/projected/282243b1-7062-446f-a987-81624327f775-kube-api-access-vh5cv\") pod \"maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj\" (UID: \"282243b1-7062-446f-a987-81624327f775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333669 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19fdb50-90db-466d-a8e4-54832a2faf0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9\" (UID: \"a19fdb50-90db-466d-a8e4-54832a2faf0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333722 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlnn2\" (UniqueName: \"kubernetes.io/projected/a19fdb50-90db-466d-a8e4-54832a2faf0b-kube-api-access-hlnn2\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9\" (UID: \"a19fdb50-90db-466d-a8e4-54832a2faf0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333762 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tqjs\" (UniqueName: \"kubernetes.io/projected/117fe53b-1b82-47cf-be04-12656d76dc65-kube-api-access-7tqjs\") pod \"maas-api-key-cleanup-29890140-kn6jd\" (UID: \"117fe53b-1b82-47cf-be04-12656d76dc65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333793 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8793d420-76d7-4900-ba05-fea167698c44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf\" (UID: \"8793d420-76d7-4900-ba05-fea167698c44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333823 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117fe53b-1b82-47cf-be04-12656d76dc65-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-kn6jd\" (UID: \"117fe53b-1b82-47cf-be04-12656d76dc65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.333858 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54b41d-d663-41cd-9280-1b67dee25ce0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479\" (UID: \"7a54b41d-d663-41cd-9280-1b67dee25ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:00.338933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.337834 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9"] Oct 31 01:00:00.434888 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.434852 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947986e0-5d7a-452c-b0ea-39d316fb6849-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826\" (UID: \"947986e0-5d7a-452c-b0ea-39d316fb6849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:00.435052 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.434899 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g767t\" (UniqueName: \"kubernetes.io/projected/947986e0-5d7a-452c-b0ea-39d316fb6849-kube-api-access-g767t\") pod \"maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826\" (UID: \"947986e0-5d7a-452c-b0ea-39d316fb6849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:00.435052 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.434937 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdlz\" (UniqueName: \"kubernetes.io/projected/8607532c-3977-469b-a040-7cd14ec8fefa-kube-api-access-spdlz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg\" (UID: \"8607532c-3977-469b-a040-7cd14ec8fefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:00.435052 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.434997 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjx2p\" (UniqueName: \"kubernetes.io/projected/7a54b41d-d663-41cd-9280-1b67dee25ce0-kube-api-access-gjx2p\") pod \"maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479\" (UID: \"7a54b41d-d663-41cd-9280-1b67dee25ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:00.435226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435059 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282243b1-7062-446f-a987-81624327f775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj\" (UID: \"282243b1-7062-446f-a987-81624327f775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:00.435226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435083 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8607532c-3977-469b-a040-7cd14ec8fefa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg\" (UID: \"8607532c-3977-469b-a040-7cd14ec8fefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:00.435226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435151 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh5cv\" (UniqueName: \"kubernetes.io/projected/282243b1-7062-446f-a987-81624327f775-kube-api-access-vh5cv\") pod \"maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj\" (UID: \"282243b1-7062-446f-a987-81624327f775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:00.435226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435215 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19fdb50-90db-466d-a8e4-54832a2faf0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9\" (UID: \"a19fdb50-90db-466d-a8e4-54832a2faf0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:00.435521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435249 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlnn2\" (UniqueName: \"kubernetes.io/projected/a19fdb50-90db-466d-a8e4-54832a2faf0b-kube-api-access-hlnn2\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9\" (UID: \"a19fdb50-90db-466d-a8e4-54832a2faf0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:00.435521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435290 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tqjs\" (UniqueName: \"kubernetes.io/projected/117fe53b-1b82-47cf-be04-12656d76dc65-kube-api-access-7tqjs\") pod \"maas-api-key-cleanup-29890140-kn6jd\" (UID: \"117fe53b-1b82-47cf-be04-12656d76dc65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:00.435521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435318 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8793d420-76d7-4900-ba05-fea167698c44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf\" (UID: \"8793d420-76d7-4900-ba05-fea167698c44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:00.435521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435391 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117fe53b-1b82-47cf-be04-12656d76dc65-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-kn6jd\" (UID: \"117fe53b-1b82-47cf-be04-12656d76dc65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:00.435521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435430 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54b41d-d663-41cd-9280-1b67dee25ce0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479\" (UID: \"7a54b41d-d663-41cd-9280-1b67dee25ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:00.435521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435488 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf2jh\" (UniqueName: \"kubernetes.io/projected/8793d420-76d7-4900-ba05-fea167698c44-kube-api-access-zf2jh\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf\" (UID: \"8793d420-76d7-4900-ba05-fea167698c44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:00.435869 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435636 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947986e0-5d7a-452c-b0ea-39d316fb6849-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826\" (UID: \"947986e0-5d7a-452c-b0ea-39d316fb6849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:00.435965 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435940 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282243b1-7062-446f-a987-81624327f775-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj\" (UID: \"282243b1-7062-446f-a987-81624327f775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:00.435998 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435962 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8607532c-3977-469b-a040-7cd14ec8fefa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg\" (UID: \"8607532c-3977-469b-a040-7cd14ec8fefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:00.435998 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.435962 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19fdb50-90db-466d-a8e4-54832a2faf0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9\" (UID: \"a19fdb50-90db-466d-a8e4-54832a2faf0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:00.436214 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.436198 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117fe53b-1b82-47cf-be04-12656d76dc65-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890140-kn6jd\" (UID: \"117fe53b-1b82-47cf-be04-12656d76dc65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:00.436411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.436388 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54b41d-d663-41cd-9280-1b67dee25ce0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479\" (UID: \"7a54b41d-d663-41cd-9280-1b67dee25ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:00.436449 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.436412 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8793d420-76d7-4900-ba05-fea167698c44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf\" (UID: \"8793d420-76d7-4900-ba05-fea167698c44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:00.495764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.495062 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdlz\" (UniqueName: \"kubernetes.io/projected/8607532c-3977-469b-a040-7cd14ec8fefa-kube-api-access-spdlz\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg\" (UID: \"8607532c-3977-469b-a040-7cd14ec8fefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:00.498943 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.498918 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlnn2\" (UniqueName: \"kubernetes.io/projected/a19fdb50-90db-466d-a8e4-54832a2faf0b-kube-api-access-hlnn2\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9\" (UID: \"a19fdb50-90db-466d-a8e4-54832a2faf0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:00.501241 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.501180 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf2jh\" (UniqueName: \"kubernetes.io/projected/8793d420-76d7-4900-ba05-fea167698c44-kube-api-access-zf2jh\") pod \"maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf\" (UID: \"8793d420-76d7-4900-ba05-fea167698c44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:00.502194 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.502139 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tqjs\" (UniqueName: \"kubernetes.io/projected/117fe53b-1b82-47cf-be04-12656d76dc65-kube-api-access-7tqjs\") pod \"maas-api-key-cleanup-29890140-kn6jd\" (UID: \"117fe53b-1b82-47cf-be04-12656d76dc65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:00.502455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.502428 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjx2p\" (UniqueName: \"kubernetes.io/projected/7a54b41d-d663-41cd-9280-1b67dee25ce0-kube-api-access-gjx2p\") pod \"maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479\" (UID: \"7a54b41d-d663-41cd-9280-1b67dee25ce0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:00.509400 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.509379 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:00.512129 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.512072 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g767t\" (UniqueName: \"kubernetes.io/projected/947986e0-5d7a-452c-b0ea-39d316fb6849-kube-api-access-g767t\") pod \"maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826\" (UID: \"947986e0-5d7a-452c-b0ea-39d316fb6849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:00.514013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.513868 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh5cv\" (UniqueName: \"kubernetes.io/projected/282243b1-7062-446f-a987-81624327f775-kube-api-access-vh5cv\") pod \"maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj\" (UID: \"282243b1-7062-446f-a987-81624327f775\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:00.518859 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.518837 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:00.530013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.524656 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:00.549016 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.547928 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:00.574563 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.574389 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:00.577079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.576981 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:00.799369 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.799345 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:00.969625 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.969547 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf"] Oct 31 01:00:00.972638 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:00.972604 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg"] Oct 31 01:00:00.974655 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:00.974518 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8793d420_76d7_4900_ba05_fea167698c44.slice/crio-fcaabe6a1116648187a10925956ed4565470db239c8af52107c2fcfb2d277420 WatchSource:0}: Error finding container fcaabe6a1116648187a10925956ed4565470db239c8af52107c2fcfb2d277420: Status 404 returned error can't find the container with id fcaabe6a1116648187a10925956ed4565470db239c8af52107c2fcfb2d277420 Oct 31 01:00:00.977731 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:00.977593 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8607532c_3977_469b_a040_7cd14ec8fefa.slice/crio-6921108bc8e64b56a1e93c6a82b215ae86fcdf8cde981d989c0dfd30e61b34d0 WatchSource:0}: Error finding container 6921108bc8e64b56a1e93c6a82b215ae86fcdf8cde981d989c0dfd30e61b34d0: Status 404 returned error can't find the container with id 6921108bc8e64b56a1e93c6a82b215ae86fcdf8cde981d989c0dfd30e61b34d0 Oct 31 01:00:01.016531 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.016504 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd"] Oct 31 01:00:01.019373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.019352 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826"] Oct 31 01:00:01.022372 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:01.022341 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117fe53b_1b82_47cf_be04_12656d76dc65.slice/crio-ee547351e4b7115d57dfe7437f7c8c45df89375eb99753f561ebfe81bc14b898 WatchSource:0}: Error finding container ee547351e4b7115d57dfe7437f7c8c45df89375eb99753f561ebfe81bc14b898: Status 404 returned error can't find the container with id ee547351e4b7115d57dfe7437f7c8c45df89375eb99753f561ebfe81bc14b898 Oct 31 01:00:01.024285 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:01.024263 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947986e0_5d7a_452c_b0ea_39d316fb6849.slice/crio-df383942ca99c46a81d7506fccffb967986e2200ea9666ac0bc7543ba742b8ac WatchSource:0}: Error finding container df383942ca99c46a81d7506fccffb967986e2200ea9666ac0bc7543ba742b8ac: Status 404 returned error can't find the container with id df383942ca99c46a81d7506fccffb967986e2200ea9666ac0bc7543ba742b8ac Oct 31 01:00:01.030327 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.030302 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj"] Oct 31 01:00:01.069915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.069891 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9"] Oct 31 01:00:01.074905 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:01.074879 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19fdb50_90db_466d_a8e4_54832a2faf0b.slice/crio-2b71734e68221ce5a132a44cff7595519b126c3ebf01e052e7a0c91d8b2e6ea9 WatchSource:0}: Error finding container 2b71734e68221ce5a132a44cff7595519b126c3ebf01e052e7a0c91d8b2e6ea9: Status 404 returned error can't find the container with id 2b71734e68221ce5a132a44cff7595519b126c3ebf01e052e7a0c91d8b2e6ea9 Oct 31 01:00:01.180304 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.180275 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479"] Oct 31 01:00:01.184851 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:01.184828 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a54b41d_d663_41cd_9280_1b67dee25ce0.slice/crio-406a9103e5f25061dae2d85190c23bae2dd337e7764bbdf12cceca5beb1bc8bc WatchSource:0}: Error finding container 406a9103e5f25061dae2d85190c23bae2dd337e7764bbdf12cceca5beb1bc8bc: Status 404 returned error can't find the container with id 406a9103e5f25061dae2d85190c23bae2dd337e7764bbdf12cceca5beb1bc8bc Oct 31 01:00:01.248465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.248438 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" event={"ID":"282243b1-7062-446f-a987-81624327f775","Type":"ContainerStarted","Data":"84338f89761740ce53a2b38300385b3c24aca86a56db3e86cd762495ddd7dc57"} Oct 31 01:00:01.249706 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.249670 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" event={"ID":"947986e0-5d7a-452c-b0ea-39d316fb6849","Type":"ContainerStarted","Data":"df383942ca99c46a81d7506fccffb967986e2200ea9666ac0bc7543ba742b8ac"} Oct 31 01:00:01.250990 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.250965 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" event={"ID":"7a54b41d-d663-41cd-9280-1b67dee25ce0","Type":"ContainerStarted","Data":"406a9103e5f25061dae2d85190c23bae2dd337e7764bbdf12cceca5beb1bc8bc"} Oct 31 01:00:01.252218 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.252190 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" event={"ID":"8607532c-3977-469b-a040-7cd14ec8fefa","Type":"ContainerStarted","Data":"6921108bc8e64b56a1e93c6a82b215ae86fcdf8cde981d989c0dfd30e61b34d0"} Oct 31 01:00:01.253730 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.253680 5719 generic.go:356] "Generic (PLEG): container finished" podID="ba77762d-91cf-43ba-89e2-e60ce150b366" containerID="769a03c14486e624e1876647ca129b7f0e431e176bb2992426640382f91495c2" exitCode=0 Oct 31 01:00:01.253814 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.253734 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" event={"ID":"ba77762d-91cf-43ba-89e2-e60ce150b366","Type":"ContainerDied","Data":"769a03c14486e624e1876647ca129b7f0e431e176bb2992426640382f91495c2"} Oct 31 01:00:01.254942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.254922 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" event={"ID":"8793d420-76d7-4900-ba05-fea167698c44","Type":"ContainerStarted","Data":"fcaabe6a1116648187a10925956ed4565470db239c8af52107c2fcfb2d277420"} Oct 31 01:00:01.256068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.256048 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" event={"ID":"117fe53b-1b82-47cf-be04-12656d76dc65","Type":"ContainerStarted","Data":"ee547351e4b7115d57dfe7437f7c8c45df89375eb99753f561ebfe81bc14b898"} Oct 31 01:00:01.257132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:01.257113 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" event={"ID":"a19fdb50-90db-466d-a8e4-54832a2faf0b","Type":"ContainerStarted","Data":"2b71734e68221ce5a132a44cff7595519b126c3ebf01e052e7a0c91d8b2e6ea9"} Oct 31 01:00:02.265315 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.265260 5719 generic.go:356] "Generic (PLEG): container finished" podID="8607532c-3977-469b-a040-7cd14ec8fefa" containerID="3969878886f0955242b187b2a3d9b30c5c5ccb9388ec12f7f1f6d36e3795f400" exitCode=60 Oct 31 01:00:02.265679 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.265343 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" event={"ID":"8607532c-3977-469b-a040-7cd14ec8fefa","Type":"ContainerDied","Data":"3969878886f0955242b187b2a3d9b30c5c5ccb9388ec12f7f1f6d36e3795f400"} Oct 31 01:00:02.265679 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.265551 5719 scope.go:122] "RemoveContainer" containerID="3969878886f0955242b187b2a3d9b30c5c5ccb9388ec12f7f1f6d36e3795f400" Oct 31 01:00:02.268264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.267224 5719 generic.go:356] "Generic (PLEG): container finished" podID="8793d420-76d7-4900-ba05-fea167698c44" containerID="5aa368f0d252104856a02b66f031d841352192525325220e835d402f746c32fb" exitCode=60 Oct 31 01:00:02.268264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.267253 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" event={"ID":"8793d420-76d7-4900-ba05-fea167698c44","Type":"ContainerDied","Data":"5aa368f0d252104856a02b66f031d841352192525325220e835d402f746c32fb"} Oct 31 01:00:02.268264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.267570 5719 scope.go:122] "RemoveContainer" containerID="5aa368f0d252104856a02b66f031d841352192525325220e835d402f746c32fb" Oct 31 01:00:02.268834 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.268806 5719 generic.go:356] "Generic (PLEG): container finished" podID="117fe53b-1b82-47cf-be04-12656d76dc65" containerID="d8c5fef762ca6e64a74ca3998af664ac737b8a676c18cf3a927ddfac70a6c9dd" exitCode=60 Oct 31 01:00:02.268908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.268890 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" event={"ID":"117fe53b-1b82-47cf-be04-12656d76dc65","Type":"ContainerDied","Data":"d8c5fef762ca6e64a74ca3998af664ac737b8a676c18cf3a927ddfac70a6c9dd"} Oct 31 01:00:02.269074 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.269060 5719 scope.go:122] "RemoveContainer" containerID="d8c5fef762ca6e64a74ca3998af664ac737b8a676c18cf3a927ddfac70a6c9dd" Oct 31 01:00:02.274642 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.274618 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" event={"ID":"a19fdb50-90db-466d-a8e4-54832a2faf0b","Type":"ContainerStarted","Data":"969e76bef280af80f267fed3174ed690e719521a72c88d5bd9a1b5ecc9538f3d"} Oct 31 01:00:02.276278 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.276262 5719 generic.go:356] "Generic (PLEG): container finished" podID="282243b1-7062-446f-a987-81624327f775" containerID="9b6f1e6dd5b750e5002ee70c70972c0fa2f00a1db9085fddf3fb7a304c3876a1" exitCode=60 Oct 31 01:00:02.276349 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.276308 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" event={"ID":"282243b1-7062-446f-a987-81624327f775","Type":"ContainerDied","Data":"9b6f1e6dd5b750e5002ee70c70972c0fa2f00a1db9085fddf3fb7a304c3876a1"} Oct 31 01:00:02.276560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.276542 5719 scope.go:122] "RemoveContainer" containerID="9b6f1e6dd5b750e5002ee70c70972c0fa2f00a1db9085fddf3fb7a304c3876a1" Oct 31 01:00:02.278229 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.278077 5719 generic.go:356] "Generic (PLEG): container finished" podID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerID="3d985f1d8e75479060711d72e6cb731bcd0bc6f4709803eafe5a7cbff3960cbb" exitCode=60 Oct 31 01:00:02.278229 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.278163 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" event={"ID":"947986e0-5d7a-452c-b0ea-39d316fb6849","Type":"ContainerDied","Data":"3d985f1d8e75479060711d72e6cb731bcd0bc6f4709803eafe5a7cbff3960cbb"} Oct 31 01:00:02.278369 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.278313 5719 scope.go:122] "RemoveContainer" containerID="3d985f1d8e75479060711d72e6cb731bcd0bc6f4709803eafe5a7cbff3960cbb" Oct 31 01:00:02.281188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.281162 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" event={"ID":"7a54b41d-d663-41cd-9280-1b67dee25ce0","Type":"ContainerStarted","Data":"1b0bdfec1c0bc14d82fc6f39fb1eec97e1f05f9f151ae05b6ec37b3d00ddf2e7"} Oct 31 01:00:02.489242 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.489215 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 01:00:02.489385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.489346 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" podStartSLOduration=2.48933339 podStartE2EDuration="2.48933339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.488165742 +0000 UTC m=+1884.024308049" watchObservedRunningTime="2026-10-31 01:00:02.48933339 +0000 UTC m=+1884.025475694" Oct 31 01:00:02.569024 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.569000 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-xks-ca\") pod \"ba77762d-91cf-43ba-89e2-e60ce150b366\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " Oct 31 01:00:02.569239 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.569220 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-openshift-ca\") pod \"ba77762d-91cf-43ba-89e2-e60ce150b366\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " Oct 31 01:00:02.569319 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.569287 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba77762d-91cf-43ba-89e2-e60ce150b366-kube-api-access-fz4mq\" (UniqueName: \"kubernetes.io/projected/ba77762d-91cf-43ba-89e2-e60ce150b366-kube-api-access-fz4mq\") pod \"ba77762d-91cf-43ba-89e2-e60ce150b366\" (UID: \"ba77762d-91cf-43ba-89e2-e60ce150b366\") " Oct 31 01:00:02.569561 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.569539 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-openshift-ca" pod "ba77762d-91cf-43ba-89e2-e60ce150b366" (UID: "ba77762d-91cf-43ba-89e2-e60ce150b366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:02.569763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.569747 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:02.571118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.571096 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-xks-ca" pod "ba77762d-91cf-43ba-89e2-e60ce150b366" (UID: "ba77762d-91cf-43ba-89e2-e60ce150b366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:00:02.571353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.571332 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba77762d-91cf-43ba-89e2-e60ce150b366-kube-api-access-fz4mq" pod "ba77762d-91cf-43ba-89e2-e60ce150b366" (UID: "ba77762d-91cf-43ba-89e2-e60ce150b366"). InnerVolumeSpecName "kube-api-access-fz4mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:02.606164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.606113 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" podStartSLOduration=2.606097268 podStartE2EDuration="2.606097268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:02.605050492 +0000 UTC m=+1884.141192797" watchObservedRunningTime="2026-10-31 01:00:02.606097268 +0000 UTC m=+1884.142239575" Oct 31 01:00:02.671034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.671001 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4mq\" (UniqueName: \"kubernetes.io/projected/ba77762d-91cf-43ba-89e2-e60ce150b366-kube-api-access-fz4mq\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:02.671034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:02.671028 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba77762d-91cf-43ba-89e2-e60ce150b366-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:03.027007 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:03.026920 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:00:03.289090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.289000 5719 generic.go:356] "Generic (PLEG): container finished" podID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerID="969e76bef280af80f267fed3174ed690e719521a72c88d5bd9a1b5ecc9538f3d" exitCode=60 Oct 31 01:00:03.289090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.289044 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" event={"ID":"a19fdb50-90db-466d-a8e4-54832a2faf0b","Type":"ContainerDied","Data":"969e76bef280af80f267fed3174ed690e719521a72c88d5bd9a1b5ecc9538f3d"} Oct 31 01:00:03.289591 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.289382 5719 scope.go:122] "RemoveContainer" containerID="969e76bef280af80f267fed3174ed690e719521a72c88d5bd9a1b5ecc9538f3d" Oct 31 01:00:03.291080 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.291056 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" event={"ID":"282243b1-7062-446f-a987-81624327f775","Type":"ContainerStarted","Data":"0195b9e3185809cc1a966fedac6367a1ce63179e5c1e359e7db028826b74ddef"} Oct 31 01:00:03.293155 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.293132 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" event={"ID":"947986e0-5d7a-452c-b0ea-39d316fb6849","Type":"ContainerStarted","Data":"6bc0afd2299af0607d5b99f960bb6e579ae96ac04665614bede1c9b5badab379"} Oct 31 01:00:03.294677 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.294655 5719 generic.go:356] "Generic (PLEG): container finished" podID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerID="1b0bdfec1c0bc14d82fc6f39fb1eec97e1f05f9f151ae05b6ec37b3d00ddf2e7" exitCode=60 Oct 31 01:00:03.294779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.294723 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" event={"ID":"7a54b41d-d663-41cd-9280-1b67dee25ce0","Type":"ContainerDied","Data":"1b0bdfec1c0bc14d82fc6f39fb1eec97e1f05f9f151ae05b6ec37b3d00ddf2e7"} Oct 31 01:00:03.295002 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.294987 5719 scope.go:122] "RemoveContainer" containerID="1b0bdfec1c0bc14d82fc6f39fb1eec97e1f05f9f151ae05b6ec37b3d00ddf2e7" Oct 31 01:00:03.296744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.296722 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" event={"ID":"8607532c-3977-469b-a040-7cd14ec8fefa","Type":"ContainerStarted","Data":"b3b2eb0a66248df24aae37c94b8e3a87b7c61870f17f3fb8578912f4b1cfc464"} Oct 31 01:00:03.298071 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.298051 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" event={"ID":"ba77762d-91cf-43ba-89e2-e60ce150b366","Type":"ContainerDied","Data":"ecd829a0b5564dea119161571d1a21d5acc7f2c4d61ce814f08a25a7b41414bf"} Oct 31 01:00:03.298161 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.298088 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecd829a0b5564dea119161571d1a21d5acc7f2c4d61ce814f08a25a7b41414bf" Oct 31 01:00:03.298161 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.298059 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd" Oct 31 01:00:03.305158 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.302713 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" event={"ID":"8793d420-76d7-4900-ba05-fea167698c44","Type":"ContainerStarted","Data":"632921529d31a6019ff266cc48f51601b568354284a5ed0640005ec98e13f727"} Oct 31 01:00:03.305238 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.305215 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" event={"ID":"117fe53b-1b82-47cf-be04-12656d76dc65","Type":"ContainerStarted","Data":"4d33dc819d9002adb97b4223632c20f4cae20aa089374bc4a16aefe3d8d2802e"} Oct 31 01:00:03.630223 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.630181 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" podStartSLOduration=3.630166966 podStartE2EDuration="3.630166966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:03.517872271 +0000 UTC m=+1885.054014577" watchObservedRunningTime="2026-10-31 01:00:03.630166966 +0000 UTC m=+1885.166309270" Oct 31 01:00:03.741123 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.741075 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" podStartSLOduration=3.741061286 podStartE2EDuration="3.741061286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:03.739435539 +0000 UTC m=+1885.275577844" watchObservedRunningTime="2026-10-31 01:00:03.741061286 +0000 UTC m=+1885.277203584" Oct 31 01:00:03.850362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.850291 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" podStartSLOduration=3.85027828 podStartE2EDuration="3.85027828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:03.849639626 +0000 UTC m=+1885.385781930" watchObservedRunningTime="2026-10-31 01:00:03.85027828 +0000 UTC m=+1885.386420585" Oct 31 01:00:03.962080 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:03.962035 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" podStartSLOduration=3.962021505 podStartE2EDuration="3.962021505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:03.95980379 +0000 UTC m=+1885.495946095" watchObservedRunningTime="2026-10-31 01:00:03.962021505 +0000 UTC m=+1885.498163807" Oct 31 01:00:04.071200 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.071146 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" podStartSLOduration=4.071132123 podStartE2EDuration="4.071132123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:04.070777882 +0000 UTC m=+1885.606920187" watchObservedRunningTime="2026-10-31 01:00:04.071132123 +0000 UTC m=+1885.607274427" Oct 31 01:00:04.313370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.313338 5719 generic.go:356] "Generic (PLEG): container finished" podID="8793d420-76d7-4900-ba05-fea167698c44" containerID="632921529d31a6019ff266cc48f51601b568354284a5ed0640005ec98e13f727" exitCode=60 Oct 31 01:00:04.314034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.313398 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" event={"ID":"8793d420-76d7-4900-ba05-fea167698c44","Type":"ContainerDied","Data":"632921529d31a6019ff266cc48f51601b568354284a5ed0640005ec98e13f727"} Oct 31 01:00:04.314034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.313472 5719 scope.go:122] "RemoveContainer" containerID="5aa368f0d252104856a02b66f031d841352192525325220e835d402f746c32fb" Oct 31 01:00:04.314034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.313754 5719 scope.go:122] "RemoveContainer" containerID="632921529d31a6019ff266cc48f51601b568354284a5ed0640005ec98e13f727" Oct 31 01:00:04.314177 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:04.314063 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf_odh-ai-gateway-infra(8793d420-76d7-4900-ba05-fea167698c44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" podUID="8793d420-76d7-4900-ba05-fea167698c44" Oct 31 01:00:04.315606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.315582 5719 generic.go:356] "Generic (PLEG): container finished" podID="117fe53b-1b82-47cf-be04-12656d76dc65" containerID="4d33dc819d9002adb97b4223632c20f4cae20aa089374bc4a16aefe3d8d2802e" exitCode=60 Oct 31 01:00:04.315768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.315667 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" event={"ID":"117fe53b-1b82-47cf-be04-12656d76dc65","Type":"ContainerDied","Data":"4d33dc819d9002adb97b4223632c20f4cae20aa089374bc4a16aefe3d8d2802e"} Oct 31 01:00:04.315907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.315893 5719 scope.go:122] "RemoveContainer" containerID="4d33dc819d9002adb97b4223632c20f4cae20aa089374bc4a16aefe3d8d2802e" Oct 31 01:00:04.316224 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:04.316202 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-kn6jd_odh-ai-gateway-infra(117fe53b-1b82-47cf-be04-12656d76dc65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" Oct 31 01:00:04.318035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.317834 5719 generic.go:356] "Generic (PLEG): container finished" podID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerID="ef709a85dbed5fc9e4a83b1c2bb1fb7ba8198f773c30f6ede7df427906fae5fb" exitCode=60 Oct 31 01:00:04.318035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.317940 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" event={"ID":"a19fdb50-90db-466d-a8e4-54832a2faf0b","Type":"ContainerDied","Data":"ef709a85dbed5fc9e4a83b1c2bb1fb7ba8198f773c30f6ede7df427906fae5fb"} Oct 31 01:00:04.318248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.318225 5719 scope.go:122] "RemoveContainer" containerID="ef709a85dbed5fc9e4a83b1c2bb1fb7ba8198f773c30f6ede7df427906fae5fb" Oct 31 01:00:04.318536 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:04.318514 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9_odh-ai-gateway-infra(a19fdb50-90db-466d-a8e4-54832a2faf0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" Oct 31 01:00:04.320037 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.320017 5719 generic.go:356] "Generic (PLEG): container finished" podID="282243b1-7062-446f-a987-81624327f775" containerID="0195b9e3185809cc1a966fedac6367a1ce63179e5c1e359e7db028826b74ddef" exitCode=60 Oct 31 01:00:04.320147 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.320060 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" event={"ID":"282243b1-7062-446f-a987-81624327f775","Type":"ContainerDied","Data":"0195b9e3185809cc1a966fedac6367a1ce63179e5c1e359e7db028826b74ddef"} Oct 31 01:00:04.320323 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.320306 5719 scope.go:122] "RemoveContainer" containerID="0195b9e3185809cc1a966fedac6367a1ce63179e5c1e359e7db028826b74ddef" Oct 31 01:00:04.320578 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:04.320557 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj_odh-ai-gateway-infra(282243b1-7062-446f-a987-81624327f775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" podUID="282243b1-7062-446f-a987-81624327f775" Oct 31 01:00:04.325433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.325413 5719 generic.go:356] "Generic (PLEG): container finished" podID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerID="6bc0afd2299af0607d5b99f960bb6e579ae96ac04665614bede1c9b5badab379" exitCode=60 Oct 31 01:00:04.325519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.325477 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" event={"ID":"947986e0-5d7a-452c-b0ea-39d316fb6849","Type":"ContainerDied","Data":"6bc0afd2299af0607d5b99f960bb6e579ae96ac04665614bede1c9b5badab379"} Oct 31 01:00:04.325686 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.325668 5719 scope.go:122] "RemoveContainer" containerID="6bc0afd2299af0607d5b99f960bb6e579ae96ac04665614bede1c9b5badab379" Oct 31 01:00:04.326022 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:04.325934 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826_odh-ai-gateway-infra(947986e0-5d7a-452c-b0ea-39d316fb6849)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" Oct 31 01:00:04.326094 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.326079 5719 scope.go:122] "RemoveContainer" containerID="d8c5fef762ca6e64a74ca3998af664ac737b8a676c18cf3a927ddfac70a6c9dd" Oct 31 01:00:04.327404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.327384 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" event={"ID":"7a54b41d-d663-41cd-9280-1b67dee25ce0","Type":"ContainerStarted","Data":"464eb003920be26c41463b9b5cc1425d6267b5d21a7bc72ef9cadb67a4ffed8d"} Oct 31 01:00:04.329328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.329310 5719 generic.go:356] "Generic (PLEG): container finished" podID="8607532c-3977-469b-a040-7cd14ec8fefa" containerID="b3b2eb0a66248df24aae37c94b8e3a87b7c61870f17f3fb8578912f4b1cfc464" exitCode=60 Oct 31 01:00:04.329405 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.329359 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" event={"ID":"8607532c-3977-469b-a040-7cd14ec8fefa","Type":"ContainerDied","Data":"b3b2eb0a66248df24aae37c94b8e3a87b7c61870f17f3fb8578912f4b1cfc464"} Oct 31 01:00:04.329535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.329524 5719 scope.go:122] "RemoveContainer" containerID="b3b2eb0a66248df24aae37c94b8e3a87b7c61870f17f3fb8578912f4b1cfc464" Oct 31 01:00:04.329739 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:04.329724 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg_odh-ai-gateway-infra(8607532c-3977-469b-a040-7cd14ec8fefa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" Oct 31 01:00:04.337524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.337506 5719 scope.go:122] "RemoveContainer" containerID="969e76bef280af80f267fed3174ed690e719521a72c88d5bd9a1b5ecc9538f3d" Oct 31 01:00:04.348206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.348186 5719 scope.go:122] "RemoveContainer" containerID="9b6f1e6dd5b750e5002ee70c70972c0fa2f00a1db9085fddf3fb7a304c3876a1" Oct 31 01:00:04.358676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.358660 5719 scope.go:122] "RemoveContainer" containerID="3d985f1d8e75479060711d72e6cb731bcd0bc6f4709803eafe5a7cbff3960cbb" Oct 31 01:00:04.370895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:04.370873 5719 scope.go:122] "RemoveContainer" containerID="3969878886f0955242b187b2a3d9b30c5c5ccb9388ec12f7f1f6d36e3795f400" Oct 31 01:00:05.338082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.338055 5719 scope.go:122] "RemoveContainer" containerID="0195b9e3185809cc1a966fedac6367a1ce63179e5c1e359e7db028826b74ddef" Oct 31 01:00:05.338542 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:05.338283 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj_odh-ai-gateway-infra(282243b1-7062-446f-a987-81624327f775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" podUID="282243b1-7062-446f-a987-81624327f775" Oct 31 01:00:05.339657 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.339638 5719 scope.go:122] "RemoveContainer" containerID="6bc0afd2299af0607d5b99f960bb6e579ae96ac04665614bede1c9b5badab379" Oct 31 01:00:05.339897 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:05.339866 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826_odh-ai-gateway-infra(947986e0-5d7a-452c-b0ea-39d316fb6849)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" Oct 31 01:00:05.341242 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.341224 5719 generic.go:356] "Generic (PLEG): container finished" podID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerID="464eb003920be26c41463b9b5cc1425d6267b5d21a7bc72ef9cadb67a4ffed8d" exitCode=60 Oct 31 01:00:05.341335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.341307 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" event={"ID":"7a54b41d-d663-41cd-9280-1b67dee25ce0","Type":"ContainerDied","Data":"464eb003920be26c41463b9b5cc1425d6267b5d21a7bc72ef9cadb67a4ffed8d"} Oct 31 01:00:05.341451 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.341403 5719 scope.go:122] "RemoveContainer" containerID="1b0bdfec1c0bc14d82fc6f39fb1eec97e1f05f9f151ae05b6ec37b3d00ddf2e7" Oct 31 01:00:05.341497 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.341475 5719 scope.go:122] "RemoveContainer" containerID="464eb003920be26c41463b9b5cc1425d6267b5d21a7bc72ef9cadb67a4ffed8d" Oct 31 01:00:05.341703 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:05.341666 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479_odh-ai-gateway-infra(7a54b41d-d663-41cd-9280-1b67dee25ce0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" Oct 31 01:00:05.343048 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.343033 5719 scope.go:122] "RemoveContainer" containerID="b3b2eb0a66248df24aae37c94b8e3a87b7c61870f17f3fb8578912f4b1cfc464" Oct 31 01:00:05.343242 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:05.343223 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg_odh-ai-gateway-infra(8607532c-3977-469b-a040-7cd14ec8fefa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" Oct 31 01:00:05.344649 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.344635 5719 scope.go:122] "RemoveContainer" containerID="632921529d31a6019ff266cc48f51601b568354284a5ed0640005ec98e13f727" Oct 31 01:00:05.344872 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:05.344848 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf_odh-ai-gateway-infra(8793d420-76d7-4900-ba05-fea167698c44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" podUID="8793d420-76d7-4900-ba05-fea167698c44" Oct 31 01:00:05.346195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.346178 5719 scope.go:122] "RemoveContainer" containerID="4d33dc819d9002adb97b4223632c20f4cae20aa089374bc4a16aefe3d8d2802e" Oct 31 01:00:05.346368 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:05.346352 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-kn6jd_odh-ai-gateway-infra(117fe53b-1b82-47cf-be04-12656d76dc65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" Oct 31 01:00:05.347898 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:05.347882 5719 scope.go:122] "RemoveContainer" containerID="ef709a85dbed5fc9e4a83b1c2bb1fb7ba8198f773c30f6ede7df427906fae5fb" Oct 31 01:00:05.348071 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:05.348055 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9_odh-ai-gateway-infra(a19fdb50-90db-466d-a8e4-54832a2faf0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" Oct 31 01:00:06.356850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:06.356823 5719 scope.go:122] "RemoveContainer" containerID="464eb003920be26c41463b9b5cc1425d6267b5d21a7bc72ef9cadb67a4ffed8d" Oct 31 01:00:06.357206 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:06.357061 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479_odh-ai-gateway-infra(7a54b41d-d663-41cd-9280-1b67dee25ce0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" Oct 31 01:00:12.180477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:12.180441 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj"] Oct 31 01:00:12.180917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:12.180622 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" containerID="cri-o://aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9" gracePeriod=30 Oct 31 01:00:12.708205 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:12.708165 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:00:12.708360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:12.708225 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:00:14.595122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:14.595092 5719 scope.go:122] "RemoveContainer" containerID="632921529d31a6019ff266cc48f51601b568354284a5ed0640005ec98e13f727" Oct 31 01:00:15.426361 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:15.426331 5719 generic.go:356] "Generic (PLEG): container finished" podID="8793d420-76d7-4900-ba05-fea167698c44" containerID="3a809ec6db7bf213016e9dc07d0768f62c6ae1d3b212424687f4b4d2b947eb53" exitCode=60 Oct 31 01:00:15.426480 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:15.426397 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" event={"ID":"8793d420-76d7-4900-ba05-fea167698c44","Type":"ContainerDied","Data":"3a809ec6db7bf213016e9dc07d0768f62c6ae1d3b212424687f4b4d2b947eb53"} Oct 31 01:00:15.426480 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:15.426454 5719 scope.go:122] "RemoveContainer" containerID="632921529d31a6019ff266cc48f51601b568354284a5ed0640005ec98e13f727" Oct 31 01:00:15.426915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:15.426895 5719 scope.go:122] "RemoveContainer" containerID="3a809ec6db7bf213016e9dc07d0768f62c6ae1d3b212424687f4b4d2b947eb53" Oct 31 01:00:15.427204 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:15.427186 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf_odh-ai-gateway-infra(8793d420-76d7-4900-ba05-fea167698c44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" podUID="8793d420-76d7-4900-ba05-fea167698c44" Oct 31 01:00:15.595244 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:15.595224 5719 scope.go:122] "RemoveContainer" containerID="ef709a85dbed5fc9e4a83b1c2bb1fb7ba8198f773c30f6ede7df427906fae5fb" Oct 31 01:00:15.595866 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:15.595786 5719 scope.go:122] "RemoveContainer" containerID="b3b2eb0a66248df24aae37c94b8e3a87b7c61870f17f3fb8578912f4b1cfc464" Oct 31 01:00:16.436535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.436509 5719 generic.go:356] "Generic (PLEG): container finished" podID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerID="97187577d3fec7425ed3de5b3786a4a07e819d2776e541bc2443ec46eb7356ed" exitCode=60 Oct 31 01:00:16.436643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.436579 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" event={"ID":"a19fdb50-90db-466d-a8e4-54832a2faf0b","Type":"ContainerDied","Data":"97187577d3fec7425ed3de5b3786a4a07e819d2776e541bc2443ec46eb7356ed"} Oct 31 01:00:16.436680 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.436641 5719 scope.go:122] "RemoveContainer" containerID="ef709a85dbed5fc9e4a83b1c2bb1fb7ba8198f773c30f6ede7df427906fae5fb" Oct 31 01:00:16.437073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.436994 5719 scope.go:122] "RemoveContainer" containerID="97187577d3fec7425ed3de5b3786a4a07e819d2776e541bc2443ec46eb7356ed" Oct 31 01:00:16.437284 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:16.437262 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9_odh-ai-gateway-infra(a19fdb50-90db-466d-a8e4-54832a2faf0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" Oct 31 01:00:16.438722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.438676 5719 generic.go:356] "Generic (PLEG): container finished" podID="8607532c-3977-469b-a040-7cd14ec8fefa" containerID="06915d923ac3720c537dd151235e505c0afc941a442589640bb0b4779293912e" exitCode=60 Oct 31 01:00:16.438823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.438728 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" event={"ID":"8607532c-3977-469b-a040-7cd14ec8fefa","Type":"ContainerDied","Data":"06915d923ac3720c537dd151235e505c0afc941a442589640bb0b4779293912e"} Oct 31 01:00:16.439074 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.439057 5719 scope.go:122] "RemoveContainer" containerID="06915d923ac3720c537dd151235e505c0afc941a442589640bb0b4779293912e" Oct 31 01:00:16.439309 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:16.439290 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg_odh-ai-gateway-infra(8607532c-3977-469b-a040-7cd14ec8fefa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" Oct 31 01:00:16.511919 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.511899 5719 scope.go:122] "RemoveContainer" containerID="b3b2eb0a66248df24aae37c94b8e3a87b7c61870f17f3fb8578912f4b1cfc464" Oct 31 01:00:16.547036 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.547013 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf"] Oct 31 01:00:16.596720 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.596091 5719 scope.go:122] "RemoveContainer" containerID="464eb003920be26c41463b9b5cc1425d6267b5d21a7bc72ef9cadb67a4ffed8d" Oct 31 01:00:16.596720 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.596454 5719 scope.go:122] "RemoveContainer" containerID="0195b9e3185809cc1a966fedac6367a1ce63179e5c1e359e7db028826b74ddef" Oct 31 01:00:16.706265 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.706244 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:16.839676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.839649 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8793d420-76d7-4900-ba05-fea167698c44-kube-api-access-zf2jh\" (UniqueName: \"kubernetes.io/projected/8793d420-76d7-4900-ba05-fea167698c44-kube-api-access-zf2jh\") pod \"8793d420-76d7-4900-ba05-fea167698c44\" (UID: \"8793d420-76d7-4900-ba05-fea167698c44\") " Oct 31 01:00:16.839812 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.839761 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8793d420-76d7-4900-ba05-fea167698c44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8793d420-76d7-4900-ba05-fea167698c44-maas-api-service-ca\") pod \"8793d420-76d7-4900-ba05-fea167698c44\" (UID: \"8793d420-76d7-4900-ba05-fea167698c44\") " Oct 31 01:00:16.840128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.840091 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8793d420-76d7-4900-ba05-fea167698c44-maas-api-service-ca" pod "8793d420-76d7-4900-ba05-fea167698c44" (UID: "8793d420-76d7-4900-ba05-fea167698c44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:16.841911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.841887 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8793d420-76d7-4900-ba05-fea167698c44-kube-api-access-zf2jh" pod "8793d420-76d7-4900-ba05-fea167698c44" (UID: "8793d420-76d7-4900-ba05-fea167698c44"). InnerVolumeSpecName "kube-api-access-zf2jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:16.941256 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.941232 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8793d420-76d7-4900-ba05-fea167698c44-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:16.941349 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:16.941259 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf2jh\" (UniqueName: \"kubernetes.io/projected/8793d420-76d7-4900-ba05-fea167698c44-kube-api-access-zf2jh\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.446859 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.446805 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" Oct 31 01:00:17.446859 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.446819 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf" event={"ID":"8793d420-76d7-4900-ba05-fea167698c44","Type":"ContainerDied","Data":"fcaabe6a1116648187a10925956ed4565470db239c8af52107c2fcfb2d277420"} Oct 31 01:00:17.447023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.446921 5719 scope.go:122] "RemoveContainer" containerID="3a809ec6db7bf213016e9dc07d0768f62c6ae1d3b212424687f4b4d2b947eb53" Oct 31 01:00:17.452308 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.452282 5719 generic.go:356] "Generic (PLEG): container finished" podID="282243b1-7062-446f-a987-81624327f775" containerID="521a6c26f1466217464667ea8ca3f49e7a0087a88828b8bf2a9756c13e6876f4" exitCode=60 Oct 31 01:00:17.452426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.452364 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" event={"ID":"282243b1-7062-446f-a987-81624327f775","Type":"ContainerDied","Data":"521a6c26f1466217464667ea8ca3f49e7a0087a88828b8bf2a9756c13e6876f4"} Oct 31 01:00:17.453381 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.453360 5719 scope.go:122] "RemoveContainer" containerID="521a6c26f1466217464667ea8ca3f49e7a0087a88828b8bf2a9756c13e6876f4" Oct 31 01:00:17.453719 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:17.453667 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj_odh-ai-gateway-infra(282243b1-7062-446f-a987-81624327f775)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" podUID="282243b1-7062-446f-a987-81624327f775" Oct 31 01:00:17.454746 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.454727 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 01:00:17.456313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.456292 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" event={"ID":"7a54b41d-d663-41cd-9280-1b67dee25ce0","Type":"ContainerStarted","Data":"ae9e463b4f11216595ca357f0a7599892f602f1cfa461c37ac659b9c07c804ed"} Oct 31 01:00:17.459355 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.459331 5719 scope.go:122] "RemoveContainer" containerID="0195b9e3185809cc1a966fedac6367a1ce63179e5c1e359e7db028826b74ddef" Oct 31 01:00:17.459898 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.459878 5719 generic.go:356] "Generic (PLEG): container finished" podID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerID="aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9" exitCode=0 Oct 31 01:00:17.460004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.459912 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" event={"ID":"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd","Type":"ContainerDied","Data":"aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9"} Oct 31 01:00:17.460004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.459943 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" event={"ID":"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd","Type":"ContainerDied","Data":"cfd4bd302a4ddc312f740e15b72157a5b4242e824b5d40a9c94cc8bb06c59006"} Oct 31 01:00:17.460004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.459975 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj" Oct 31 01:00:17.473236 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.473220 5719 scope.go:122] "RemoveContainer" containerID="aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9" Oct 31 01:00:17.481904 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.481883 5719 scope.go:122] "RemoveContainer" containerID="aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9" Oct 31 01:00:17.482189 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:17.482163 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9\": container with ID starting with aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9 not found: ID does not exist" containerID="aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9" Oct 31 01:00:17.482253 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.482200 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9"} err="failed to get container status \"aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9\": rpc error: code = NotFound desc = could not find container \"aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9\": container with ID starting with aaddc7fc2fbc56c9f005f53afced0e9343f30c8545c3da69d09e90a9ebede6b9 not found: ID does not exist" Oct 31 01:00:17.546427 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546397 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-certs\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.546549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546440 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-token\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-token\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.546549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546506 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-credential-socket\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.546549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546529 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-envoy\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.546779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546607 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-data\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.546779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546628 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-certs" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:17.546779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546669 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-podinfo\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.546779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546748 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istiod-ca-cert\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.546963 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546796 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-credential-socket" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:17.546963 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546821 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-socket\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.546963 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546862 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-data" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:17.546963 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.546925 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-kube-api-access-qsdww\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-kube-api-access-qsdww\") pod \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\" (UID: \"01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd\") " Oct 31 01:00:17.547147 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.547035 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-socket" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:17.547199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.547154 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istiod-ca-cert" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:17.547659 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.547635 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.547763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.547663 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.547763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.547677 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.547763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.547705 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.547763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.547719 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.548884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.548858 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-token" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:17.549328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.549304 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-envoy" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:17.549537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.549519 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-podinfo" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:17.549920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.549901 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-kube-api-access-qsdww" pod "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" (UID: "01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd"). InnerVolumeSpecName "kube-api-access-qsdww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:17.560907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.560887 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9"] Oct 31 01:00:17.602468 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.602387 5719 scope.go:122] "RemoveContainer" containerID="4d33dc819d9002adb97b4223632c20f4cae20aa089374bc4a16aefe3d8d2802e" Oct 31 01:00:17.603196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.603174 5719 scope.go:122] "RemoveContainer" containerID="6bc0afd2299af0607d5b99f960bb6e579ae96ac04665614bede1c9b5badab379" Oct 31 01:00:17.663742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.652143 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.663742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.652177 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.663742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.652193 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.663742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.656842 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsdww\" (UniqueName: \"kubernetes.io/projected/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd-kube-api-access-qsdww\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.672555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.672531 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg"] Oct 31 01:00:17.722031 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.722002 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf"] Oct 31 01:00:17.729127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.728992 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8bec73-29890140-mb6sf"] Oct 31 01:00:17.764878 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.764811 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:17.863126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.863096 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19fdb50-90db-466d-a8e4-54832a2faf0b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19fdb50-90db-466d-a8e4-54832a2faf0b-maas-api-service-ca\") pod \"a19fdb50-90db-466d-a8e4-54832a2faf0b\" (UID: \"a19fdb50-90db-466d-a8e4-54832a2faf0b\") " Oct 31 01:00:17.863533 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.863480 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19fdb50-90db-466d-a8e4-54832a2faf0b-maas-api-service-ca" pod "a19fdb50-90db-466d-a8e4-54832a2faf0b" (UID: "a19fdb50-90db-466d-a8e4-54832a2faf0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:17.863533 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.863504 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19fdb50-90db-466d-a8e4-54832a2faf0b-kube-api-access-hlnn2\" (UniqueName: \"kubernetes.io/projected/a19fdb50-90db-466d-a8e4-54832a2faf0b-kube-api-access-hlnn2\") pod \"a19fdb50-90db-466d-a8e4-54832a2faf0b\" (UID: \"a19fdb50-90db-466d-a8e4-54832a2faf0b\") " Oct 31 01:00:17.863994 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.863976 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19fdb50-90db-466d-a8e4-54832a2faf0b-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.866295 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.866267 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19fdb50-90db-466d-a8e4-54832a2faf0b-kube-api-access-hlnn2" pod "a19fdb50-90db-466d-a8e4-54832a2faf0b" (UID: "a19fdb50-90db-466d-a8e4-54832a2faf0b"). InnerVolumeSpecName "kube-api-access-hlnn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:17.895245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.895221 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:17.966035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.965292 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8607532c-3977-469b-a040-7cd14ec8fefa-kube-api-access-spdlz\" (UniqueName: \"kubernetes.io/projected/8607532c-3977-469b-a040-7cd14ec8fefa-kube-api-access-spdlz\") pod \"8607532c-3977-469b-a040-7cd14ec8fefa\" (UID: \"8607532c-3977-469b-a040-7cd14ec8fefa\") " Oct 31 01:00:17.966035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.965427 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8607532c-3977-469b-a040-7cd14ec8fefa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8607532c-3977-469b-a040-7cd14ec8fefa-maas-api-service-ca\") pod \"8607532c-3977-469b-a040-7cd14ec8fefa\" (UID: \"8607532c-3977-469b-a040-7cd14ec8fefa\") " Oct 31 01:00:17.966425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.966396 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8607532c-3977-469b-a040-7cd14ec8fefa-maas-api-service-ca" pod "8607532c-3977-469b-a040-7cd14ec8fefa" (UID: "8607532c-3977-469b-a040-7cd14ec8fefa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:17.966593 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.966574 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlnn2\" (UniqueName: \"kubernetes.io/projected/a19fdb50-90db-466d-a8e4-54832a2faf0b-kube-api-access-hlnn2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.966661 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.966600 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8607532c-3977-469b-a040-7cd14ec8fefa-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:17.971033 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:17.970972 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8607532c-3977-469b-a040-7cd14ec8fefa-kube-api-access-spdlz" pod "8607532c-3977-469b-a040-7cd14ec8fefa" (UID: "8607532c-3977-469b-a040-7cd14ec8fefa"). InnerVolumeSpecName "kube-api-access-spdlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:18.068046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.068018 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdlz\" (UniqueName: \"kubernetes.io/projected/8607532c-3977-469b-a040-7cd14ec8fefa-kube-api-access-spdlz\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:18.216247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.216214 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj"] Oct 31 01:00:18.252259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.252199 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qc9cj"] Oct 31 01:00:18.471616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.471553 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" event={"ID":"a19fdb50-90db-466d-a8e4-54832a2faf0b","Type":"ContainerDied","Data":"2b71734e68221ce5a132a44cff7595519b126c3ebf01e052e7a0c91d8b2e6ea9"} Oct 31 01:00:18.471998 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.471631 5719 scope.go:122] "RemoveContainer" containerID="97187577d3fec7425ed3de5b3786a4a07e819d2776e541bc2443ec46eb7356ed" Oct 31 01:00:18.471998 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.471757 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9" Oct 31 01:00:18.476595 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.476572 5719 generic.go:356] "Generic (PLEG): container finished" podID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerID="50754978110db766d1699138bb298a6216019041e2f569fa8409bb789c4a0c41" exitCode=60 Oct 31 01:00:18.476759 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.476647 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" event={"ID":"947986e0-5d7a-452c-b0ea-39d316fb6849","Type":"ContainerDied","Data":"50754978110db766d1699138bb298a6216019041e2f569fa8409bb789c4a0c41"} Oct 31 01:00:18.477088 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.477070 5719 scope.go:122] "RemoveContainer" containerID="50754978110db766d1699138bb298a6216019041e2f569fa8409bb789c4a0c41" Oct 31 01:00:18.477318 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:18.477299 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826_odh-ai-gateway-infra(947986e0-5d7a-452c-b0ea-39d316fb6849)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" Oct 31 01:00:18.478810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.478791 5719 generic.go:356] "Generic (PLEG): container finished" podID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerID="ae9e463b4f11216595ca357f0a7599892f602f1cfa461c37ac659b9c07c804ed" exitCode=60 Oct 31 01:00:18.478896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.478864 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" event={"ID":"7a54b41d-d663-41cd-9280-1b67dee25ce0","Type":"ContainerDied","Data":"ae9e463b4f11216595ca357f0a7599892f602f1cfa461c37ac659b9c07c804ed"} Oct 31 01:00:18.479264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.479236 5719 scope.go:122] "RemoveContainer" containerID="ae9e463b4f11216595ca357f0a7599892f602f1cfa461c37ac659b9c07c804ed" Oct 31 01:00:18.479516 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:18.479495 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479_odh-ai-gateway-infra(7a54b41d-d663-41cd-9280-1b67dee25ce0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" Oct 31 01:00:18.480890 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.480862 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" Oct 31 01:00:18.481851 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.481785 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg" event={"ID":"8607532c-3977-469b-a040-7cd14ec8fefa","Type":"ContainerDied","Data":"6921108bc8e64b56a1e93c6a82b215ae86fcdf8cde981d989c0dfd30e61b34d0"} Oct 31 01:00:18.484576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.484561 5719 scope.go:122] "RemoveContainer" containerID="6bc0afd2299af0607d5b99f960bb6e579ae96ac04665614bede1c9b5badab379" Oct 31 01:00:18.485863 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.485832 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" event={"ID":"117fe53b-1b82-47cf-be04-12656d76dc65","Type":"ContainerStarted","Data":"66c5a1304b5a7ffa76d24c6375f0b60f9f6f320423156eba9be31bb08a69641a"} Oct 31 01:00:18.497478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.497459 5719 scope.go:122] "RemoveContainer" containerID="464eb003920be26c41463b9b5cc1425d6267b5d21a7bc72ef9cadb67a4ffed8d" Oct 31 01:00:18.508952 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.508936 5719 scope.go:122] "RemoveContainer" containerID="06915d923ac3720c537dd151235e505c0afc941a442589640bb0b4779293912e" Oct 31 01:00:18.613680 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.613645 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" path="/var/lib/kubelet/pods/01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd/volumes" Oct 31 01:00:18.614139 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.614121 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8793d420-76d7-4900-ba05-fea167698c44" path="/var/lib/kubelet/pods/8793d420-76d7-4900-ba05-fea167698c44/volumes" Oct 31 01:00:18.624530 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.624505 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479"] Oct 31 01:00:18.743801 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.743769 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9"] Oct 31 01:00:18.750399 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.750373 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890140-4wbq9"] Oct 31 01:00:18.855059 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.855031 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj"] Oct 31 01:00:18.857499 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.857473 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg"] Oct 31 01:00:18.863806 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:18.863784 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890140-jtkfg"] Oct 31 01:00:19.037442 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.037419 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:19.180715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.180617 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/282243b1-7062-446f-a987-81624327f775-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282243b1-7062-446f-a987-81624327f775-maas-api-service-ca\") pod \"282243b1-7062-446f-a987-81624327f775\" (UID: \"282243b1-7062-446f-a987-81624327f775\") " Oct 31 01:00:19.180870 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.180854 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/282243b1-7062-446f-a987-81624327f775-kube-api-access-vh5cv\" (UniqueName: \"kubernetes.io/projected/282243b1-7062-446f-a987-81624327f775-kube-api-access-vh5cv\") pod \"282243b1-7062-446f-a987-81624327f775\" (UID: \"282243b1-7062-446f-a987-81624327f775\") " Oct 31 01:00:19.180975 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.180952 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/282243b1-7062-446f-a987-81624327f775-maas-api-service-ca" pod "282243b1-7062-446f-a987-81624327f775" (UID: "282243b1-7062-446f-a987-81624327f775"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.181230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.181211 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/282243b1-7062-446f-a987-81624327f775-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.183239 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.183214 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282243b1-7062-446f-a987-81624327f775-kube-api-access-vh5cv" pod "282243b1-7062-446f-a987-81624327f775" (UID: "282243b1-7062-446f-a987-81624327f775"). InnerVolumeSpecName "kube-api-access-vh5cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.282925 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.282894 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh5cv\" (UniqueName: \"kubernetes.io/projected/282243b1-7062-446f-a987-81624327f775-kube-api-access-vh5cv\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.494251 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.494176 5719 generic.go:356] "Generic (PLEG): container finished" podID="117fe53b-1b82-47cf-be04-12656d76dc65" containerID="66c5a1304b5a7ffa76d24c6375f0b60f9f6f320423156eba9be31bb08a69641a" exitCode=60 Oct 31 01:00:19.494251 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.494221 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" event={"ID":"117fe53b-1b82-47cf-be04-12656d76dc65","Type":"ContainerDied","Data":"66c5a1304b5a7ffa76d24c6375f0b60f9f6f320423156eba9be31bb08a69641a"} Oct 31 01:00:19.494484 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.494274 5719 scope.go:122] "RemoveContainer" containerID="4d33dc819d9002adb97b4223632c20f4cae20aa089374bc4a16aefe3d8d2802e" Oct 31 01:00:19.494619 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.494599 5719 scope.go:122] "RemoveContainer" containerID="66c5a1304b5a7ffa76d24c6375f0b60f9f6f320423156eba9be31bb08a69641a" Oct 31 01:00:19.494930 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:19.494903 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890140-kn6jd_odh-ai-gateway-infra(117fe53b-1b82-47cf-be04-12656d76dc65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" Oct 31 01:00:19.497082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.497061 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" event={"ID":"282243b1-7062-446f-a987-81624327f775","Type":"ContainerDied","Data":"84338f89761740ce53a2b38300385b3c24aca86a56db3e86cd762495ddd7dc57"} Oct 31 01:00:19.497082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.497076 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj" Oct 31 01:00:19.507352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.507331 5719 scope.go:122] "RemoveContainer" containerID="521a6c26f1466217464667ea8ca3f49e7a0087a88828b8bf2a9756c13e6876f4" Oct 31 01:00:19.643632 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.643604 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:19.645682 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.645660 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826"] Oct 31 01:00:19.788425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.788399 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a54b41d-d663-41cd-9280-1b67dee25ce0-kube-api-access-gjx2p\" (UniqueName: \"kubernetes.io/projected/7a54b41d-d663-41cd-9280-1b67dee25ce0-kube-api-access-gjx2p\") pod \"7a54b41d-d663-41cd-9280-1b67dee25ce0\" (UID: \"7a54b41d-d663-41cd-9280-1b67dee25ce0\") " Oct 31 01:00:19.788577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.788480 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a54b41d-d663-41cd-9280-1b67dee25ce0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54b41d-d663-41cd-9280-1b67dee25ce0-maas-api-service-ca\") pod \"7a54b41d-d663-41cd-9280-1b67dee25ce0\" (UID: \"7a54b41d-d663-41cd-9280-1b67dee25ce0\") " Oct 31 01:00:19.788851 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.788826 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a54b41d-d663-41cd-9280-1b67dee25ce0-maas-api-service-ca" pod "7a54b41d-d663-41cd-9280-1b67dee25ce0" (UID: "7a54b41d-d663-41cd-9280-1b67dee25ce0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.788999 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.788983 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a54b41d-d663-41cd-9280-1b67dee25ce0-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.790532 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.790506 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a54b41d-d663-41cd-9280-1b67dee25ce0-kube-api-access-gjx2p" pod "7a54b41d-d663-41cd-9280-1b67dee25ce0" (UID: "7a54b41d-d663-41cd-9280-1b67dee25ce0"). InnerVolumeSpecName "kube-api-access-gjx2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.791365 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.791343 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:19.827111 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.827092 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj"] Oct 31 01:00:19.833530 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.833511 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f6979f-29890140-spfrj"] Oct 31 01:00:19.889430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.889409 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/947986e0-5d7a-452c-b0ea-39d316fb6849-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947986e0-5d7a-452c-b0ea-39d316fb6849-maas-api-service-ca\") pod \"947986e0-5d7a-452c-b0ea-39d316fb6849\" (UID: \"947986e0-5d7a-452c-b0ea-39d316fb6849\") " Oct 31 01:00:19.889521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.889510 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/947986e0-5d7a-452c-b0ea-39d316fb6849-kube-api-access-g767t\" (UniqueName: \"kubernetes.io/projected/947986e0-5d7a-452c-b0ea-39d316fb6849-kube-api-access-g767t\") pod \"947986e0-5d7a-452c-b0ea-39d316fb6849\" (UID: \"947986e0-5d7a-452c-b0ea-39d316fb6849\") " Oct 31 01:00:19.889780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.889760 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947986e0-5d7a-452c-b0ea-39d316fb6849-maas-api-service-ca" pod "947986e0-5d7a-452c-b0ea-39d316fb6849" (UID: "947986e0-5d7a-452c-b0ea-39d316fb6849"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:19.889840 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.889785 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjx2p\" (UniqueName: \"kubernetes.io/projected/7a54b41d-d663-41cd-9280-1b67dee25ce0-kube-api-access-gjx2p\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.891540 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.891521 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947986e0-5d7a-452c-b0ea-39d316fb6849-kube-api-access-g767t" pod "947986e0-5d7a-452c-b0ea-39d316fb6849" (UID: "947986e0-5d7a-452c-b0ea-39d316fb6849"). InnerVolumeSpecName "kube-api-access-g767t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:19.991044 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.991008 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g767t\" (UniqueName: \"kubernetes.io/projected/947986e0-5d7a-452c-b0ea-39d316fb6849-kube-api-access-g767t\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:19.991044 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:19.991038 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/947986e0-5d7a-452c-b0ea-39d316fb6849-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.095722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.095670 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd"] Oct 31 01:00:20.510676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.510608 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" Oct 31 01:00:20.510676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.510611 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826" event={"ID":"947986e0-5d7a-452c-b0ea-39d316fb6849","Type":"ContainerDied","Data":"df383942ca99c46a81d7506fccffb967986e2200ea9666ac0bc7543ba742b8ac"} Oct 31 01:00:20.510913 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.510755 5719 scope.go:122] "RemoveContainer" containerID="50754978110db766d1699138bb298a6216019041e2f569fa8409bb789c4a0c41" Oct 31 01:00:20.512045 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.512031 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" Oct 31 01:00:20.512127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.512075 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479" event={"ID":"7a54b41d-d663-41cd-9280-1b67dee25ce0","Type":"ContainerDied","Data":"406a9103e5f25061dae2d85190c23bae2dd337e7764bbdf12cceca5beb1bc8bc"} Oct 31 01:00:20.522494 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.522478 5719 scope.go:122] "RemoveContainer" containerID="ae9e463b4f11216595ca357f0a7599892f602f1cfa461c37ac659b9c07c804ed" Oct 31 01:00:20.610029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.610000 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="282243b1-7062-446f-a987-81624327f775" path="/var/lib/kubelet/pods/282243b1-7062-446f-a987-81624327f775/volumes" Oct 31 01:00:20.610412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.610395 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" path="/var/lib/kubelet/pods/8607532c-3977-469b-a040-7cd14ec8fefa/volumes" Oct 31 01:00:20.610795 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.610780 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" path="/var/lib/kubelet/pods/a19fdb50-90db-466d-a8e4-54832a2faf0b/volumes" Oct 31 01:00:20.644391 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.644371 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:20.668970 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.668950 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479"] Oct 31 01:00:20.674976 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.674954 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-cb2b8a-29890140-mj479"] Oct 31 01:00:20.800258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.800206 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117fe53b-1b82-47cf-be04-12656d76dc65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117fe53b-1b82-47cf-be04-12656d76dc65-maas-api-service-ca\") pod \"117fe53b-1b82-47cf-be04-12656d76dc65\" (UID: \"117fe53b-1b82-47cf-be04-12656d76dc65\") " Oct 31 01:00:20.800343 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.800311 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117fe53b-1b82-47cf-be04-12656d76dc65-kube-api-access-7tqjs\" (UniqueName: \"kubernetes.io/projected/117fe53b-1b82-47cf-be04-12656d76dc65-kube-api-access-7tqjs\") pod \"117fe53b-1b82-47cf-be04-12656d76dc65\" (UID: \"117fe53b-1b82-47cf-be04-12656d76dc65\") " Oct 31 01:00:20.800580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.800560 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117fe53b-1b82-47cf-be04-12656d76dc65-maas-api-service-ca" pod "117fe53b-1b82-47cf-be04-12656d76dc65" (UID: "117fe53b-1b82-47cf-be04-12656d76dc65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:20.800756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.800743 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/117fe53b-1b82-47cf-be04-12656d76dc65-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:20.802406 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.802390 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117fe53b-1b82-47cf-be04-12656d76dc65-kube-api-access-7tqjs" pod "117fe53b-1b82-47cf-be04-12656d76dc65" (UID: "117fe53b-1b82-47cf-be04-12656d76dc65"). InnerVolumeSpecName "kube-api-access-7tqjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:20.891793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.891768 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826"] Oct 31 01:00:20.895763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.895739 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ac952a-29890140-n5826"] Oct 31 01:00:20.901405 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:20.901376 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tqjs\" (UniqueName: \"kubernetes.io/projected/117fe53b-1b82-47cf-be04-12656d76dc65-kube-api-access-7tqjs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:21.523959 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:21.523924 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" event={"ID":"117fe53b-1b82-47cf-be04-12656d76dc65","Type":"ContainerDied","Data":"ee547351e4b7115d57dfe7437f7c8c45df89375eb99753f561ebfe81bc14b898"} Oct 31 01:00:21.524154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:21.523968 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd" Oct 31 01:00:21.524154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:21.524027 5719 scope.go:122] "RemoveContainer" containerID="66c5a1304b5a7ffa76d24c6375f0b60f9f6f320423156eba9be31bb08a69641a" Oct 31 01:00:21.681059 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:21.681018 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd"] Oct 31 01:00:21.685239 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:21.685215 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890140-kn6jd"] Oct 31 01:00:22.603806 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:22.603776 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" path="/var/lib/kubelet/pods/117fe53b-1b82-47cf-be04-12656d76dc65/volumes" Oct 31 01:00:22.604171 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:22.604157 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" path="/var/lib/kubelet/pods/7a54b41d-d663-41cd-9280-1b67dee25ce0/volumes" Oct 31 01:00:22.604541 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:22.604526 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" path="/var/lib/kubelet/pods/947986e0-5d7a-452c-b0ea-39d316fb6849/volumes" Oct 31 01:00:24.455828 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:24.455798 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9"] Oct 31 01:00:24.456255 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:24.456017 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" containerID="cri-o://87785b4d8476637f60b486e5f95bfdc7e8ea954807c7e1682e080f83c8f85b7f" gracePeriod=30 Oct 31 01:00:24.504968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:24.504932 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8"] Oct 31 01:00:24.505206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:24.505178 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" containerID="cri-o://c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10" gracePeriod=30 Oct 31 01:00:26.925433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:26.925402 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:00:26.925823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:26.925465 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:00:29.398033 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.397986 5719 patch_prober.go:29] interesting pod/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:00:29.398457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.398048 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:00:29.588000 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.587971 5719 generic.go:356] "Generic (PLEG): container finished" podID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerID="87785b4d8476637f60b486e5f95bfdc7e8ea954807c7e1682e080f83c8f85b7f" exitCode=0 Oct 31 01:00:29.588176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.588055 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" event={"ID":"bd537fc9-02b4-4392-a012-5d6ce626670f","Type":"ContainerDied","Data":"87785b4d8476637f60b486e5f95bfdc7e8ea954807c7e1682e080f83c8f85b7f"} Oct 31 01:00:29.777910 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.777885 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 01:00:29.809740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.809683 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 01:00:29.810050 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810035 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd537fc9-02b4-4392-a012-5d6ce626670f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd537fc9-02b4-4392-a012-5d6ce626670f-istiod-ca-cert\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810063 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-token\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-token\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810160 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810143 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-socket\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810236 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810216 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-credential-socket\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810361 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810303 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-certs\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810360 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-envoy\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810410 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-kube-api-access-dl8px\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-kube-api-access-dl8px\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810457 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-data\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810465 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-socket" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.810537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810495 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd537fc9-02b4-4392-a012-5d6ce626670f-istiod-ca-cert" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:29.810674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810535 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-credential-socket" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.810674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810571 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-podinfo\") pod \"bd537fc9-02b4-4392-a012-5d6ce626670f\" (UID: \"bd537fc9-02b4-4392-a012-5d6ce626670f\") " Oct 31 01:00:29.810674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810581 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-certs" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.810825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.810783 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-data" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.811316 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.811255 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd537fc9-02b4-4392-a012-5d6ce626670f-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.811374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.811322 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.811374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.811337 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.811374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.811350 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.811374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.811363 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.812799 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.812778 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-kube-api-access-dl8px" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "kube-api-access-dl8px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:29.812867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.812797 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-podinfo" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:29.813030 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.813005 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-envoy" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.813030 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.813019 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-token" pod "bd537fc9-02b4-4392-a012-5d6ce626670f" (UID: "bd537fc9-02b4-4392-a012-5d6ce626670f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:29.911825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.911801 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-socket\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.911924 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.911833 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-credential-socket\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.911924 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.911890 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-kube-api-access-nz27s\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-kube-api-access-nz27s\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.911924 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.911910 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e9d5c6-bc61-4014-913e-e9a948da2594-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9d5c6-bc61-4014-913e-e9a948da2594-istiod-ca-cert\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.912084 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.911981 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-token\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.912084 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.911999 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-envoy\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.912084 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912047 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-socket" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.912084 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912078 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-data\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.912317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912130 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-podinfo\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.912317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912149 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-credential-socket" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.912317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912159 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-certs\") pod \"b2e9d5c6-bc61-4014-913e-e9a948da2594\" (UID: \"b2e9d5c6-bc61-4014-913e-e9a948da2594\") " Oct 31 01:00:29.912317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912298 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e9d5c6-bc61-4014-913e-e9a948da2594-istiod-ca-cert" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:00:29.912667 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912364 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-data" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.912667 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912381 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-certs" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.912897 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912878 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.912966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912904 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.912966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912922 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.912966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912937 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.912966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912951 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.912966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912965 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e9d5c6-bc61-4014-913e-e9a948da2594-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.913236 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912980 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.913236 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.912995 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl8px\" (UniqueName: \"kubernetes.io/projected/bd537fc9-02b4-4392-a012-5d6ce626670f-kube-api-access-dl8px\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.913236 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.913009 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd537fc9-02b4-4392-a012-5d6ce626670f-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:29.914147 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.914123 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-envoy" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:00:29.914231 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.914123 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-token" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:29.914271 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.914232 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-kube-api-access-nz27s" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "kube-api-access-nz27s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:00:29.914325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:29.914308 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-podinfo" pod "b2e9d5c6-bc61-4014-913e-e9a948da2594" (UID: "b2e9d5c6-bc61-4014-913e-e9a948da2594"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:00:30.014290 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.014218 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:30.014290 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.014246 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz27s\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-kube-api-access-nz27s\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:30.014290 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.014256 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:30.014290 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.014264 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e9d5c6-bc61-4014-913e-e9a948da2594-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:00:30.611082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.611048 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" Oct 31 01:00:30.612144 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.612117 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9" event={"ID":"bd537fc9-02b4-4392-a012-5d6ce626670f","Type":"ContainerDied","Data":"5f943e195bc5c9f1ade290485c6c8f22a7d72a4496e4ce82111d4a8cd39c6a66"} Oct 31 01:00:30.612317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.612218 5719 scope.go:122] "RemoveContainer" containerID="87785b4d8476637f60b486e5f95bfdc7e8ea954807c7e1682e080f83c8f85b7f" Oct 31 01:00:30.612805 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.612778 5719 generic.go:356] "Generic (PLEG): container finished" podID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerID="c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10" exitCode=0 Oct 31 01:00:30.612929 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.612836 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" Oct 31 01:00:30.612929 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.612921 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" event={"ID":"b2e9d5c6-bc61-4014-913e-e9a948da2594","Type":"ContainerDied","Data":"c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10"} Oct 31 01:00:30.613035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.612957 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8" event={"ID":"b2e9d5c6-bc61-4014-913e-e9a948da2594","Type":"ContainerDied","Data":"ed7bd75be11d6929cd14ec279d1fb9e1292998acf340a3725bb86abeee3130a0"} Oct 31 01:00:30.624344 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.624323 5719 scope.go:122] "RemoveContainer" containerID="c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10" Oct 31 01:00:30.633228 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.633211 5719 scope.go:122] "RemoveContainer" containerID="c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10" Oct 31 01:00:30.633519 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:30.633500 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10\": container with ID starting with c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10 not found: ID does not exist" containerID="c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10" Oct 31 01:00:30.633573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.633528 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10"} err="failed to get container status \"c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10\": rpc error: code = NotFound desc = could not find container \"c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10\": container with ID starting with c193cd91895c7c7e5ed02bf1714f3d166e3fab84067122a7f3955ccee9af3f10 not found: ID does not exist" Oct 31 01:00:30.817570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.817546 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9"] Oct 31 01:00:30.865001 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.864973 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-dnbt9"] Oct 31 01:00:30.929165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.929137 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8"] Oct 31 01:00:30.933240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:30.933215 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8bec73-openshift-default-5bb9f8ccd5-zp7t8"] Oct 31 01:00:32.602879 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:32.602845 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" path="/var/lib/kubelet/pods/b2e9d5c6-bc61-4014-913e-e9a948da2594/volumes" Oct 31 01:00:32.603310 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:32.603297 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" path="/var/lib/kubelet/pods/bd537fc9-02b4-4392-a012-5d6ce626670f/volumes" Oct 31 01:00:33.009967 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:00:33.009889 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:00:40.818122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:40.818088 5719 scope.go:122] "RemoveContainer" containerID="3d0ec279052bd2d31f701a5665135c65c088a93b44fd3d47e2c4d7f9c3bed32f" Oct 31 01:00:44.688230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:44.688201 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:00:56.744923 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.744891 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v"] Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745347 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745359 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745368 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745373 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745391 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745396 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745404 5719 cpu_manager.go:414] "containerMap: removing container" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745408 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745414 5719 cpu_manager.go:414] "containerMap: removing container" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745419 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745426 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.745434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745430 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745450 5719 cpu_manager.go:414] "containerMap: removing container" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745454 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745463 5719 cpu_manager.go:414] "containerMap: removing container" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745468 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745485 5719 cpu_manager.go:414] "containerMap: removing container" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745491 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745501 5719 cpu_manager.go:414] "containerMap: removing container" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745505 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745514 5719 cpu_manager.go:414] "containerMap: removing container" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745518 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745525 5719 cpu_manager.go:414] "containerMap: removing container" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745528 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745534 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745538 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745547 5719 cpu_manager.go:414] "containerMap: removing container" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745552 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745561 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ba77762d-91cf-43ba-89e2-e60ce150b366" containerName="revoke-keys" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745568 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba77762d-91cf-43ba-89e2-e60ce150b366" containerName="revoke-keys" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745581 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745585 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745592 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745597 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745607 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745611 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745620 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745625 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745741 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745754 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745761 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745772 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745780 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745785 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745789 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745795 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745801 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745811 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745819 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745828 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e9d5c6-bc61-4014-913e-e9a948da2594" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745834 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745840 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745845 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745851 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd537fc9-02b4-4392-a012-5d6ce626670f" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745856 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745862 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fb9cdd-4cd8-4ecd-b29c-69a6605ff1fd" containerName="istio-proxy" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745867 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba77762d-91cf-43ba-89e2-e60ce150b366" containerName="revoke-keys" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745873 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745878 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.745895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.745887 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19fdb50-90db-466d-a8e4-54832a2faf0b" containerName="cleanup" Oct 31 01:00:56.748311 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.748295 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:56.798565 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.798531 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:00:56.846463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.846424 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:00:56.846640 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.846474 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:00:56.846640 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.846526 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:00:56.848901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.848880 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v"] Oct 31 01:00:56.872778 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.872754 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6"] Oct 31 01:00:56.873222 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873211 5719 cpu_manager.go:414] "containerMap: removing container" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.873267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873224 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117fe53b-1b82-47cf-be04-12656d76dc65" containerName="cleanup" Oct 31 01:00:56.873267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873257 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.873267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873261 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a54b41d-d663-41cd-9280-1b67dee25ce0" containerName="cleanup" Oct 31 01:00:56.873359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873270 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.873359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873275 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.873359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873282 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.873359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873286 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.873359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873298 5719 cpu_manager.go:414] "containerMap: removing container" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.873359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873304 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.873359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873312 5719 cpu_manager.go:414] "containerMap: removing container" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.873359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873317 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="947986e0-5d7a-452c-b0ea-39d316fb6849" containerName="cleanup" Oct 31 01:00:56.873592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873400 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8793d420-76d7-4900-ba05-fea167698c44" containerName="cleanup" Oct 31 01:00:56.873592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873406 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8607532c-3977-469b-a040-7cd14ec8fefa" containerName="cleanup" Oct 31 01:00:56.873592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.873412 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="282243b1-7062-446f-a987-81624327f775" containerName="cleanup" Oct 31 01:00:56.875783 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.875769 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:56.899992 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.899973 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:56.900107 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.900026 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:56.900158 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.900114 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj7ht\" (UniqueName: \"kubernetes.io/projected/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-kube-api-access-fj7ht\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:56.925950 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.925923 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-31ae40-dockercfg-w6crv\"" Oct 31 01:00:56.925950 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.925941 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-31ae40\"" Oct 31 01:00:56.932142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:56.932121 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6"] Oct 31 01:00:57.002766 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.001685 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:57.002766 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.001825 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:57.012861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.003054 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6\" (UID: \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:57.012861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.003127 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44sht\" (UniqueName: \"kubernetes.io/projected/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-kube-api-access-44sht\") pod \"payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6\" (UID: \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:57.012861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.003169 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj7ht\" (UniqueName: \"kubernetes.io/projected/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-kube-api-access-fj7ht\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:57.012861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.006619 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:57.012861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.010178 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:57.050297 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.050250 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm"] Oct 31 01:00:57.054265 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.054240 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:57.067801 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.067774 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj7ht\" (UniqueName: \"kubernetes.io/projected/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-kube-api-access-fj7ht\") pod \"maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:57.104303 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.104279 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6\" (UID: \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:57.104427 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.104323 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44sht\" (UniqueName: \"kubernetes.io/projected/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-kube-api-access-44sht\") pod \"payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6\" (UID: \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:57.104990 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.104966 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6\" (UID: \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:57.114755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.114726 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm"] Oct 31 01:00:57.163295 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.163259 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44sht\" (UniqueName: \"kubernetes.io/projected/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-kube-api-access-44sht\") pod \"payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6\" (UID: \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:57.187825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.187795 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:57.205940 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.205905 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htr6\" (UniqueName: \"kubernetes.io/projected/4b3465ad-042f-4aba-99f9-2286ba50b1ef-kube-api-access-8htr6\") pod \"payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm\" (UID: \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\") " pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:57.206132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.206109 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3465ad-042f-4aba-99f9-2286ba50b1ef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm\" (UID: \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\") " pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:57.307453 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.307383 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3465ad-042f-4aba-99f9-2286ba50b1ef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm\" (UID: \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\") " pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:57.307591 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.307505 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htr6\" (UniqueName: \"kubernetes.io/projected/4b3465ad-042f-4aba-99f9-2286ba50b1ef-kube-api-access-8htr6\") pod \"payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm\" (UID: \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\") " pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:57.308080 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.308059 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3465ad-042f-4aba-99f9-2286ba50b1ef-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm\" (UID: \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\") " pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:57.364460 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.364430 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:00:57.366353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.366330 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htr6\" (UniqueName: \"kubernetes.io/projected/4b3465ad-042f-4aba-99f9-2286ba50b1ef-kube-api-access-8htr6\") pod \"payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm\" (UID: \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\") " pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:57.368404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.368382 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:57.580407 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.580374 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6"] Oct 31 01:00:57.585509 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:57.585479 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fdbb1b_6682_4bfe_8f46_af5a41fafa36.slice/crio-6ecc8b9855da41e8aba1a6d2ca35fd668617840a5084ee84a3225b22241a5777 WatchSource:0}: Error finding container 6ecc8b9855da41e8aba1a6d2ca35fd668617840a5084ee84a3225b22241a5777: Status 404 returned error can't find the container with id 6ecc8b9855da41e8aba1a6d2ca35fd668617840a5084ee84a3225b22241a5777 Oct 31 01:00:57.740394 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.740368 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v"] Oct 31 01:00:57.743095 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.743077 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm"] Oct 31 01:00:57.745771 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:57.745745 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda127fdcd_4b57_4266_87be_7cea5b8ccd4c.slice/crio-9a691b9f2e8a069958eba5102aef73b3083abdb92315f232e5f0ff7475c3d842 WatchSource:0}: Error finding container 9a691b9f2e8a069958eba5102aef73b3083abdb92315f232e5f0ff7475c3d842: Status 404 returned error can't find the container with id 9a691b9f2e8a069958eba5102aef73b3083abdb92315f232e5f0ff7475c3d842 Oct 31 01:00:57.747753 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:00:57.747731 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3465ad_042f_4aba_99f9_2286ba50b1ef.slice/crio-1ed151bce25988e957f408b4a297ce60e80199d71d941eea54027095fee7d54f WatchSource:0}: Error finding container 1ed151bce25988e957f408b4a297ce60e80199d71d941eea54027095fee7d54f: Status 404 returned error can't find the container with id 1ed151bce25988e957f408b4a297ce60e80199d71d941eea54027095fee7d54f Oct 31 01:00:57.769888 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.769865 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" event={"ID":"4b3465ad-042f-4aba-99f9-2286ba50b1ef","Type":"ContainerStarted","Data":"1ed151bce25988e957f408b4a297ce60e80199d71d941eea54027095fee7d54f"} Oct 31 01:00:57.771211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.771183 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" event={"ID":"a127fdcd-4b57-4266-87be-7cea5b8ccd4c","Type":"ContainerStarted","Data":"9a691b9f2e8a069958eba5102aef73b3083abdb92315f232e5f0ff7475c3d842"} Oct 31 01:00:57.772742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.772717 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" event={"ID":"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36","Type":"ContainerStarted","Data":"26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc"} Oct 31 01:00:57.772830 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.772761 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" event={"ID":"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36","Type":"ContainerStarted","Data":"6ecc8b9855da41e8aba1a6d2ca35fd668617840a5084ee84a3225b22241a5777"} Oct 31 01:00:57.772830 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.772824 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:00:57.884643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:57.884567 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" podStartSLOduration=1.8845548399999998 podStartE2EDuration="1.88455484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:57.884050653 +0000 UTC m=+1939.420192957" watchObservedRunningTime="2026-10-31 01:00:57.88455484 +0000 UTC m=+1939.420697145" Oct 31 01:00:58.779761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:58.779727 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" event={"ID":"4b3465ad-042f-4aba-99f9-2286ba50b1ef","Type":"ContainerStarted","Data":"42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff"} Oct 31 01:00:58.780199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:58.779877 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:00:58.781344 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:58.781315 5719 generic.go:356] "Generic (PLEG): container finished" podID="a127fdcd-4b57-4266-87be-7cea5b8ccd4c" containerID="da454a574b2304c95146a1c8c40e10a8031b1568e2af9815f56fccf5ad04a696" exitCode=0 Oct 31 01:00:58.781480 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:58.781425 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" event={"ID":"a127fdcd-4b57-4266-87be-7cea5b8ccd4c","Type":"ContainerDied","Data":"da454a574b2304c95146a1c8c40e10a8031b1568e2af9815f56fccf5ad04a696"} Oct 31 01:00:58.892893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:58.892848 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" podStartSLOduration=1.8928361169999999 podStartE2EDuration="1.892836117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:58.891000425 +0000 UTC m=+1940.427142730" watchObservedRunningTime="2026-10-31 01:00:58.892836117 +0000 UTC m=+1940.428978443" Oct 31 01:00:59.949370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:00:59.949344 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:01:00.036170 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.036129 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-xks-ca\") pod \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " Oct 31 01:01:00.036325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.036300 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-kube-api-access-fj7ht\" (UniqueName: \"kubernetes.io/projected/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-kube-api-access-fj7ht\") pod \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " Oct 31 01:01:00.036385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.036360 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-openshift-ca\") pod \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\" (UID: \"a127fdcd-4b57-4266-87be-7cea5b8ccd4c\") " Oct 31 01:01:00.036745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.036685 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-openshift-ca" pod "a127fdcd-4b57-4266-87be-7cea5b8ccd4c" (UID: "a127fdcd-4b57-4266-87be-7cea5b8ccd4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:00.038479 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.038459 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-xks-ca" pod "a127fdcd-4b57-4266-87be-7cea5b8ccd4c" (UID: "a127fdcd-4b57-4266-87be-7cea5b8ccd4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:00.038584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.038502 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-kube-api-access-fj7ht" pod "a127fdcd-4b57-4266-87be-7cea5b8ccd4c" (UID: "a127fdcd-4b57-4266-87be-7cea5b8ccd4c"). InnerVolumeSpecName "kube-api-access-fj7ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:00.137992 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.137958 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:00.137992 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.137988 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:00.137992 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.137998 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj7ht\" (UniqueName: \"kubernetes.io/projected/a127fdcd-4b57-4266-87be-7cea5b8ccd4c-kube-api-access-fj7ht\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:00.798246 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.798215 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" Oct 31 01:01:00.798406 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.798258 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v" event={"ID":"a127fdcd-4b57-4266-87be-7cea5b8ccd4c","Type":"ContainerDied","Data":"9a691b9f2e8a069958eba5102aef73b3083abdb92315f232e5f0ff7475c3d842"} Oct 31 01:01:00.798406 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:00.798307 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a691b9f2e8a069958eba5102aef73b3083abdb92315f232e5f0ff7475c3d842" Oct 31 01:01:03.010359 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:01:03.010326 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:01:08.784277 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:08.784248 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:01:09.793675 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:09.793645 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:01:11.231154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:11.230982 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd"] Oct 31 01:01:11.277655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:11.277625 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f6979f-ndbzd"] Oct 31 01:01:12.602916 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:12.602885 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba77762d-91cf-43ba-89e2-e60ce150b366" path="/var/lib/kubelet/pods/ba77762d-91cf-43ba-89e2-e60ce150b366/volumes" Oct 31 01:01:27.021400 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.021364 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh"] Oct 31 01:01:27.021873 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.021859 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a127fdcd-4b57-4266-87be-7cea5b8ccd4c" containerName="revoke-keys" Oct 31 01:01:27.021921 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.021874 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a127fdcd-4b57-4266-87be-7cea5b8ccd4c" containerName="revoke-keys" Oct 31 01:01:27.022001 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.021992 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a127fdcd-4b57-4266-87be-7cea5b8ccd4c" containerName="revoke-keys" Oct 31 01:01:27.025277 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.025260 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.075785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.075753 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:01:27.123321 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.123286 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:27.123556 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.123538 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:01:27.123609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.123549 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:01:27.125978 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.125959 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh"] Oct 31 01:01:27.148650 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.148620 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26076112-5134-4aad-a830-6c047abca282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.148796 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.148755 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26076112-5134-4aad-a830-6c047abca282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.148883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.148798 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8b7d\" (UniqueName: \"kubernetes.io/projected/26076112-5134-4aad-a830-6c047abca282-kube-api-access-s8b7d\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.250181 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.250148 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8b7d\" (UniqueName: \"kubernetes.io/projected/26076112-5134-4aad-a830-6c047abca282-kube-api-access-s8b7d\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.250332 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.250274 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26076112-5134-4aad-a830-6c047abca282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.250399 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.250346 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26076112-5134-4aad-a830-6c047abca282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.250999 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.250976 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26076112-5134-4aad-a830-6c047abca282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.252814 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.252795 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26076112-5134-4aad-a830-6c047abca282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.311391 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.311313 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8b7d\" (UniqueName: \"kubernetes.io/projected/26076112-5134-4aad-a830-6c047abca282-kube-api-access-s8b7d\") pod \"maas-api-revoke-sub-106dbcf1-bgkrh\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.338227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.338199 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:27.708752 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.708719 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh"] Oct 31 01:01:27.714399 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:01:27.714366 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26076112_5134_4aad_a830_6c047abca282.slice/crio-895bcaa6ae682245e38ceaf59e8b8e0915c4b82358fb690aa0c18a982026bfbd WatchSource:0}: Error finding container 895bcaa6ae682245e38ceaf59e8b8e0915c4b82358fb690aa0c18a982026bfbd: Status 404 returned error can't find the container with id 895bcaa6ae682245e38ceaf59e8b8e0915c4b82358fb690aa0c18a982026bfbd Oct 31 01:01:27.970025 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:27.969947 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" event={"ID":"26076112-5134-4aad-a830-6c047abca282","Type":"ContainerStarted","Data":"895bcaa6ae682245e38ceaf59e8b8e0915c4b82358fb690aa0c18a982026bfbd"} Oct 31 01:01:28.977766 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:28.977733 5719 generic.go:356] "Generic (PLEG): container finished" podID="26076112-5134-4aad-a830-6c047abca282" containerID="92d628500c53f7862870a6bdd5300884f0a92e5b446bcf15ebf7cf016e14f900" exitCode=0 Oct 31 01:01:28.978169 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:28.977798 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" event={"ID":"26076112-5134-4aad-a830-6c047abca282","Type":"ContainerDied","Data":"92d628500c53f7862870a6bdd5300884f0a92e5b446bcf15ebf7cf016e14f900"} Oct 31 01:01:30.123218 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.123186 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:30.180549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.180521 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26076112-5134-4aad-a830-6c047abca282-kube-api-access-s8b7d\" (UniqueName: \"kubernetes.io/projected/26076112-5134-4aad-a830-6c047abca282-kube-api-access-s8b7d\") pod \"26076112-5134-4aad-a830-6c047abca282\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " Oct 31 01:01:30.180738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.180604 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26076112-5134-4aad-a830-6c047abca282-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26076112-5134-4aad-a830-6c047abca282-maas-api-xks-ca\") pod \"26076112-5134-4aad-a830-6c047abca282\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " Oct 31 01:01:30.180798 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.180746 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26076112-5134-4aad-a830-6c047abca282-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26076112-5134-4aad-a830-6c047abca282-maas-api-openshift-ca\") pod \"26076112-5134-4aad-a830-6c047abca282\" (UID: \"26076112-5134-4aad-a830-6c047abca282\") " Oct 31 01:01:30.181323 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.181180 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26076112-5134-4aad-a830-6c047abca282-maas-api-openshift-ca" pod "26076112-5134-4aad-a830-6c047abca282" (UID: "26076112-5134-4aad-a830-6c047abca282"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:30.182709 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.182671 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26076112-5134-4aad-a830-6c047abca282-kube-api-access-s8b7d" pod "26076112-5134-4aad-a830-6c047abca282" (UID: "26076112-5134-4aad-a830-6c047abca282"). InnerVolumeSpecName "kube-api-access-s8b7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:30.182801 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.182672 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26076112-5134-4aad-a830-6c047abca282-maas-api-xks-ca" pod "26076112-5134-4aad-a830-6c047abca282" (UID: "26076112-5134-4aad-a830-6c047abca282"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:30.282496 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.282428 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8b7d\" (UniqueName: \"kubernetes.io/projected/26076112-5134-4aad-a830-6c047abca282-kube-api-access-s8b7d\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:30.282496 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.282454 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26076112-5134-4aad-a830-6c047abca282-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:30.282496 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.282463 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26076112-5134-4aad-a830-6c047abca282-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:30.991257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.991177 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" Oct 31 01:01:30.991257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.991189 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh" event={"ID":"26076112-5134-4aad-a830-6c047abca282","Type":"ContainerDied","Data":"895bcaa6ae682245e38ceaf59e8b8e0915c4b82358fb690aa0c18a982026bfbd"} Oct 31 01:01:30.991257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:30.991235 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="895bcaa6ae682245e38ceaf59e8b8e0915c4b82358fb690aa0c18a982026bfbd" Oct 31 01:01:32.553211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:32.553181 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:01:33.009811 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:01:33.009776 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:01:37.191439 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.191408 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx"] Oct 31 01:01:37.191960 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.191940 5719 cpu_manager.go:414] "containerMap: removing container" podUID="26076112-5134-4aad-a830-6c047abca282" containerName="revoke-keys" Oct 31 01:01:37.191960 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.191957 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26076112-5134-4aad-a830-6c047abca282" containerName="revoke-keys" Oct 31 01:01:37.192127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.192051 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="26076112-5134-4aad-a830-6c047abca282" containerName="revoke-keys" Oct 31 01:01:37.195292 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.195274 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.280240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.280198 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.280447 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.280404 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.280447 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.280432 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69h8d\" (UniqueName: \"kubernetes.io/projected/70c85efa-42ee-421a-ac5c-392d8dac9afa-kube-api-access-69h8d\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.296056 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.296026 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:01:37.296056 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.296028 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:01:37.296266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.296086 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:01:37.296266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.296135 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:01:37.298520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.298501 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx"] Oct 31 01:01:37.381899 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.381867 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.382068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.381905 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69h8d\" (UniqueName: \"kubernetes.io/projected/70c85efa-42ee-421a-ac5c-392d8dac9afa-kube-api-access-69h8d\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.382068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.382000 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.382663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.382646 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.384288 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.384268 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.440998 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.440974 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69h8d\" (UniqueName: \"kubernetes.io/projected/70c85efa-42ee-421a-ac5c-392d8dac9afa-kube-api-access-69h8d\") pod \"maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.509212 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.509153 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:37.903733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:37.903673 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx"] Oct 31 01:01:37.908958 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:01:37.908932 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c85efa_42ee_421a_ac5c_392d8dac9afa.slice/crio-99dfb38b2adfab4ba4858e22cf3049f5569199ab822dc4b12f5b29c381f341fb WatchSource:0}: Error finding container 99dfb38b2adfab4ba4858e22cf3049f5569199ab822dc4b12f5b29c381f341fb: Status 404 returned error can't find the container with id 99dfb38b2adfab4ba4858e22cf3049f5569199ab822dc4b12f5b29c381f341fb Oct 31 01:01:38.036247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:38.036209 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" event={"ID":"70c85efa-42ee-421a-ac5c-392d8dac9afa","Type":"ContainerStarted","Data":"99dfb38b2adfab4ba4858e22cf3049f5569199ab822dc4b12f5b29c381f341fb"} Oct 31 01:01:39.043570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:39.043536 5719 generic.go:356] "Generic (PLEG): container finished" podID="70c85efa-42ee-421a-ac5c-392d8dac9afa" containerID="fdebb7ba6cc9d4100af9678fe651a3bcc4a2112e9fc8ec2e97da9505fccab4ca" exitCode=0 Oct 31 01:01:39.043963 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:39.043616 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" event={"ID":"70c85efa-42ee-421a-ac5c-392d8dac9afa","Type":"ContainerDied","Data":"fdebb7ba6cc9d4100af9678fe651a3bcc4a2112e9fc8ec2e97da9505fccab4ca"} Oct 31 01:01:40.198948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.198921 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:40.224532 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.224502 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-openshift-ca\") pod \"70c85efa-42ee-421a-ac5c-392d8dac9afa\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " Oct 31 01:01:40.224678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.224574 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c85efa-42ee-421a-ac5c-392d8dac9afa-kube-api-access-69h8d\" (UniqueName: \"kubernetes.io/projected/70c85efa-42ee-421a-ac5c-392d8dac9afa-kube-api-access-69h8d\") pod \"70c85efa-42ee-421a-ac5c-392d8dac9afa\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " Oct 31 01:01:40.224678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.224643 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-xks-ca\") pod \"70c85efa-42ee-421a-ac5c-392d8dac9afa\" (UID: \"70c85efa-42ee-421a-ac5c-392d8dac9afa\") " Oct 31 01:01:40.224903 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.224866 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-openshift-ca" pod "70c85efa-42ee-421a-ac5c-392d8dac9afa" (UID: "70c85efa-42ee-421a-ac5c-392d8dac9afa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:01:40.225015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.224982 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:40.226888 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.226868 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-xks-ca" pod "70c85efa-42ee-421a-ac5c-392d8dac9afa" (UID: "70c85efa-42ee-421a-ac5c-392d8dac9afa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:01:40.226954 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.226870 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c85efa-42ee-421a-ac5c-392d8dac9afa-kube-api-access-69h8d" pod "70c85efa-42ee-421a-ac5c-392d8dac9afa" (UID: "70c85efa-42ee-421a-ac5c-392d8dac9afa"). InnerVolumeSpecName "kube-api-access-69h8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:01:40.325788 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.325759 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69h8d\" (UniqueName: \"kubernetes.io/projected/70c85efa-42ee-421a-ac5c-392d8dac9afa-kube-api-access-69h8d\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:40.325788 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.325785 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c85efa-42ee-421a-ac5c-392d8dac9afa-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:01:40.882668 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.882591 5719 scope.go:122] "RemoveContainer" containerID="51989ae0965a1e49795c11a6a2114b7e0e18cf32618ce3682cced4a46b33ecae" Oct 31 01:01:40.892115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:40.892085 5719 scope.go:122] "RemoveContainer" containerID="0c96304160b8908ccbd1464f34735258af78cbf9293f23a125fc4a1361a4d64e" Oct 31 01:01:41.058095 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:41.058058 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" Oct 31 01:01:41.058095 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:41.058087 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx" event={"ID":"70c85efa-42ee-421a-ac5c-392d8dac9afa","Type":"ContainerDied","Data":"99dfb38b2adfab4ba4858e22cf3049f5569199ab822dc4b12f5b29c381f341fb"} Oct 31 01:01:41.058303 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:41.058192 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99dfb38b2adfab4ba4858e22cf3049f5569199ab822dc4b12f5b29c381f341fb" Oct 31 01:01:42.544289 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.544256 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf"] Oct 31 01:01:42.544808 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.544792 5719 cpu_manager.go:414] "containerMap: removing container" podUID="70c85efa-42ee-421a-ac5c-392d8dac9afa" containerName="revoke-keys" Oct 31 01:01:42.544867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.544809 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c85efa-42ee-421a-ac5c-392d8dac9afa" containerName="revoke-keys" Oct 31 01:01:42.544945 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.544935 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c85efa-42ee-421a-ac5c-392d8dac9afa" containerName="revoke-keys" Oct 31 01:01:42.549682 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.549657 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:42.652488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.652455 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9e4801\"" Oct 31 01:01:42.652488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.652455 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9e4801-dockercfg-lcbd7\"" Oct 31 01:01:42.654804 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.654784 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf"] Oct 31 01:01:42.655383 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.655368 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6fq\" (UniqueName: \"kubernetes.io/projected/5a34ee25-d63f-4ed2-8a45-304105ec08ee-kube-api-access-fh6fq\") pod \"payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf\" (UID: \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:42.655480 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.655453 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a34ee25-d63f-4ed2-8a45-304105ec08ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf\" (UID: \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:42.756503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.756456 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6fq\" (UniqueName: \"kubernetes.io/projected/5a34ee25-d63f-4ed2-8a45-304105ec08ee-kube-api-access-fh6fq\") pod \"payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf\" (UID: \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:42.756745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.756576 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a34ee25-d63f-4ed2-8a45-304105ec08ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf\" (UID: \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:42.757435 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.757412 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a34ee25-d63f-4ed2-8a45-304105ec08ee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf\" (UID: \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:42.815555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.815482 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6fq\" (UniqueName: \"kubernetes.io/projected/5a34ee25-d63f-4ed2-8a45-304105ec08ee-kube-api-access-fh6fq\") pod \"payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf\" (UID: \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:42.863452 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:42.863420 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:43.234979 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:43.234902 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf"] Oct 31 01:01:43.240315 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:01:43.240282 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a34ee25_d63f_4ed2_8a45_304105ec08ee.slice/crio-cc62dded3a5ac6886d4da53ff767dae39ca7e0c4676c14109922285867b5ca32 WatchSource:0}: Error finding container cc62dded3a5ac6886d4da53ff767dae39ca7e0c4676c14109922285867b5ca32: Status 404 returned error can't find the container with id cc62dded3a5ac6886d4da53ff767dae39ca7e0c4676c14109922285867b5ca32 Oct 31 01:01:44.079510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:44.079473 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" event={"ID":"5a34ee25-d63f-4ed2-8a45-304105ec08ee","Type":"ContainerStarted","Data":"bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10"} Oct 31 01:01:44.080079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:44.079535 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" event={"ID":"5a34ee25-d63f-4ed2-8a45-304105ec08ee","Type":"ContainerStarted","Data":"cc62dded3a5ac6886d4da53ff767dae39ca7e0c4676c14109922285867b5ca32"} Oct 31 01:01:44.080079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:44.079649 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:01:44.192606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:44.192548 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" podStartSLOduration=2.19253316 podStartE2EDuration="2.19253316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:44.191335739 +0000 UTC m=+1985.727478043" watchObservedRunningTime="2026-10-31 01:01:44.19253316 +0000 UTC m=+1985.728675465" Oct 31 01:01:55.089539 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:01:55.089459 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:02:03.010574 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:02:03.010535 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:02:15.004461 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:15.004423 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p"] Oct 31 01:02:15.004977 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:15.004659 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" containerID="cri-o://a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70" gracePeriod=30 Oct 31 01:02:15.053622 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:15.053591 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx"] Oct 31 01:02:15.057065 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:15.057038 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b647c3-97cxx"] Oct 31 01:02:15.552323 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:15.552292 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:02:16.602374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:16.602344 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c85efa-42ee-421a-ac5c-392d8dac9afa" path="/var/lib/kubelet/pods/70c85efa-42ee-421a-ac5c-392d8dac9afa/volumes" Oct 31 01:02:19.251765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:19.251728 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sf86p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:02:19.252285 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:19.251790 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:02:20.261959 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.261918 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 01:02:20.299315 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.299234 5719 generic.go:356] "Generic (PLEG): container finished" podID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerID="a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70" exitCode=0 Oct 31 01:02:20.299315 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.299281 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" Oct 31 01:02:20.299510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.299318 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" event={"ID":"22f5caf1-12b8-4298-a800-5cb5811455ab","Type":"ContainerDied","Data":"a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70"} Oct 31 01:02:20.299510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.299368 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p" event={"ID":"22f5caf1-12b8-4298-a800-5cb5811455ab","Type":"ContainerDied","Data":"3e1af41b804bfe913a8f4cdaf5b2334e68e7c2a2d744aa54cd7d22deb191f4e9"} Oct 31 01:02:20.299510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.299389 5719 scope.go:122] "RemoveContainer" containerID="a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70" Oct 31 01:02:20.309300 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.309281 5719 scope.go:122] "RemoveContainer" containerID="a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70" Oct 31 01:02:20.309616 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:02:20.309590 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70\": container with ID starting with a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70 not found: ID does not exist" containerID="a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70" Oct 31 01:02:20.309678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.309626 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70"} err="failed to get container status \"a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70\": rpc error: code = NotFound desc = could not find container \"a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70\": container with ID starting with a9631430a140a28e095a0cbca232cde22d333ac65c5d7199441568eac0a30d70 not found: ID does not exist" Oct 31 01:02:20.387434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387396 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-kube-api-access-xmxrz\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-kube-api-access-xmxrz\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.387629 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387495 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-token\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-token\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.387629 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387525 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-certs\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.387629 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387614 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-socket\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.387847 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387659 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f5caf1-12b8-4298-a800-5cb5811455ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f5caf1-12b8-4298-a800-5cb5811455ab-istiod-ca-cert\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.387847 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387763 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-podinfo\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.387847 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387822 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-envoy\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.388004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387854 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-credential-socket\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.388004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387907 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-socket" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:20.388004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.387918 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-certs" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:20.388249 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388177 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-credential-socket" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:20.388358 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388327 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-data\") pod \"22f5caf1-12b8-4298-a800-5cb5811455ab\" (UID: \"22f5caf1-12b8-4298-a800-5cb5811455ab\") " Oct 31 01:02:20.388651 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388601 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-data" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:20.388651 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388620 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f5caf1-12b8-4298-a800-5cb5811455ab-istiod-ca-cert" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:20.388888 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388871 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.388962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388892 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.388962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388905 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.388962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388914 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.388962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.388922 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22f5caf1-12b8-4298-a800-5cb5811455ab-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.390164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.390141 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-envoy" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:20.390273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.390249 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-kube-api-access-xmxrz" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "kube-api-access-xmxrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:20.390376 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.390354 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-token" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:20.390571 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.390551 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-podinfo" pod "22f5caf1-12b8-4298-a800-5cb5811455ab" (UID: "22f5caf1-12b8-4298-a800-5cb5811455ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:20.489616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.489584 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.489616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.489610 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.489616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.489620 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxrz\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-kube-api-access-xmxrz\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.489884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.489630 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22f5caf1-12b8-4298-a800-5cb5811455ab-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:20.723413 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.723379 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p"] Oct 31 01:02:20.726983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:20.726944 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sf86p"] Oct 31 01:02:22.602150 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:22.602114 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" path="/var/lib/kubelet/pods/22f5caf1-12b8-4298-a800-5cb5811455ab/volumes" Oct 31 01:02:23.422217 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.422179 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t"] Oct 31 01:02:23.422765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.422745 5719 cpu_manager.go:414] "containerMap: removing container" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" Oct 31 01:02:23.422869 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.422765 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" Oct 31 01:02:23.422951 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.422934 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f5caf1-12b8-4298-a800-5cb5811455ab" containerName="istio-proxy" Oct 31 01:02:23.427407 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.427385 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.478682 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.478648 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:02:23.481920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.481892 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t"] Oct 31 01:02:23.524241 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.524205 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.524431 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.524364 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.524431 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.524417 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wls2d\" (UniqueName: \"kubernetes.io/projected/0ff3b0fe-841c-4c48-a0ea-12209349547a-kube-api-access-wls2d\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.526501 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.526483 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:02:23.526610 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.526514 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:02:23.526610 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.526579 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:02:23.625760 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.625724 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.626204 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.625840 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.626204 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.625882 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wls2d\" (UniqueName: \"kubernetes.io/projected/0ff3b0fe-841c-4c48-a0ea-12209349547a-kube-api-access-wls2d\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.626333 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.626288 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.628320 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.628298 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.685060 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.684987 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wls2d\" (UniqueName: \"kubernetes.io/projected/0ff3b0fe-841c-4c48-a0ea-12209349547a-kube-api-access-wls2d\") pod \"maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:23.749492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:23.749462 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:24.159931 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:24.159897 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t"] Oct 31 01:02:24.164728 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:02:24.164682 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff3b0fe_841c_4c48_a0ea_12209349547a.slice/crio-1f907546a3f1c9d8e88a28edd75d9d066d8f0e26a2c1f8ee8000695a9915f5ea WatchSource:0}: Error finding container 1f907546a3f1c9d8e88a28edd75d9d066d8f0e26a2c1f8ee8000695a9915f5ea: Status 404 returned error can't find the container with id 1f907546a3f1c9d8e88a28edd75d9d066d8f0e26a2c1f8ee8000695a9915f5ea Oct 31 01:02:24.325556 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:24.325526 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" event={"ID":"0ff3b0fe-841c-4c48-a0ea-12209349547a","Type":"ContainerStarted","Data":"1f907546a3f1c9d8e88a28edd75d9d066d8f0e26a2c1f8ee8000695a9915f5ea"} Oct 31 01:02:25.333542 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:25.333464 5719 generic.go:356] "Generic (PLEG): container finished" podID="0ff3b0fe-841c-4c48-a0ea-12209349547a" containerID="63e38c6e47a602bb7d9c8f9cb136c06aa700f35998362c6f0b1f4bc5106d9ac5" exitCode=0 Oct 31 01:02:25.333925 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:25.333542 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" event={"ID":"0ff3b0fe-841c-4c48-a0ea-12209349547a","Type":"ContainerDied","Data":"63e38c6e47a602bb7d9c8f9cb136c06aa700f35998362c6f0b1f4bc5106d9ac5"} Oct 31 01:02:26.501313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.501287 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:26.660082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.660054 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-xks-ca\") pod \"0ff3b0fe-841c-4c48-a0ea-12209349547a\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " Oct 31 01:02:26.660082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.660089 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff3b0fe-841c-4c48-a0ea-12209349547a-kube-api-access-wls2d\" (UniqueName: \"kubernetes.io/projected/0ff3b0fe-841c-4c48-a0ea-12209349547a-kube-api-access-wls2d\") pod \"0ff3b0fe-841c-4c48-a0ea-12209349547a\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " Oct 31 01:02:26.660353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.660191 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-openshift-ca\") pod \"0ff3b0fe-841c-4c48-a0ea-12209349547a\" (UID: \"0ff3b0fe-841c-4c48-a0ea-12209349547a\") " Oct 31 01:02:26.660627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.660601 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-openshift-ca" pod "0ff3b0fe-841c-4c48-a0ea-12209349547a" (UID: "0ff3b0fe-841c-4c48-a0ea-12209349547a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:26.662291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.662263 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-xks-ca" pod "0ff3b0fe-841c-4c48-a0ea-12209349547a" (UID: "0ff3b0fe-841c-4c48-a0ea-12209349547a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:02:26.662384 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.662351 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff3b0fe-841c-4c48-a0ea-12209349547a-kube-api-access-wls2d" pod "0ff3b0fe-841c-4c48-a0ea-12209349547a" (UID: "0ff3b0fe-841c-4c48-a0ea-12209349547a"). InnerVolumeSpecName "kube-api-access-wls2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:26.762325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.762229 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:26.762325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.762271 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ff3b0fe-841c-4c48-a0ea-12209349547a-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:26.762325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:26.762289 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wls2d\" (UniqueName: \"kubernetes.io/projected/0ff3b0fe-841c-4c48-a0ea-12209349547a-kube-api-access-wls2d\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:27.345320 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:27.345289 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" Oct 31 01:02:27.345496 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:27.345324 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t" event={"ID":"0ff3b0fe-841c-4c48-a0ea-12209349547a","Type":"ContainerDied","Data":"1f907546a3f1c9d8e88a28edd75d9d066d8f0e26a2c1f8ee8000695a9915f5ea"} Oct 31 01:02:27.345496 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:27.345373 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f907546a3f1c9d8e88a28edd75d9d066d8f0e26a2c1f8ee8000695a9915f5ea" Oct 31 01:02:30.008744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:30.008706 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9"] Oct 31 01:02:30.011864 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:30.008961 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" containerID="cri-o://226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be" gracePeriod=30 Oct 31 01:02:31.918920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:31.918888 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v"] Oct 31 01:02:31.966397 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:31.966370 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ac952a-d677v"] Oct 31 01:02:32.602271 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:32.602239 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a127fdcd-4b57-4266-87be-7cea5b8ccd4c" path="/var/lib/kubelet/pods/a127fdcd-4b57-4266-87be-7cea5b8ccd4c/volumes" Oct 31 01:02:33.009811 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:02:33.009739 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:02:35.271737 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.271451 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 01:02:35.361522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361487 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-credential-socket\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.361790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361531 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-data\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.361790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361578 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39984c71-6603-47a3-9344-7407edf3bcd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39984c71-6603-47a3-9344-7407edf3bcd2-istio-podinfo\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.361790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361647 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-kube-api-access-4kdfc\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-kube-api-access-4kdfc\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.361790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361746 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-socket\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.362023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361801 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-istio-token\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-istio-token\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.362023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361826 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-certs\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.362023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361826 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-credential-socket" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.362023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361853 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-data" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.362023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361911 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-envoy\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.362023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.361973 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39984c71-6603-47a3-9344-7407edf3bcd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39984c71-6603-47a3-9344-7407edf3bcd2-istiod-ca-cert\") pod \"39984c71-6603-47a3-9344-7407edf3bcd2\" (UID: \"39984c71-6603-47a3-9344-7407edf3bcd2\") " Oct 31 01:02:35.362338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.362101 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-socket" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.362338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.362122 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-certs" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.362457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.362431 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39984c71-6603-47a3-9344-7407edf3bcd2-istiod-ca-cert" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:35.362671 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.362653 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.362773 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.362678 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.362773 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.362722 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.362773 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.362738 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.364067 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.364047 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-kube-api-access-4kdfc" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "kube-api-access-4kdfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:35.364146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.364130 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-istio-token" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:35.364555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.364536 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39984c71-6603-47a3-9344-7407edf3bcd2-istio-podinfo" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:35.364619 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.364550 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-envoy" pod "39984c71-6603-47a3-9344-7407edf3bcd2" (UID: "39984c71-6603-47a3-9344-7407edf3bcd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:35.393095 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.393069 5719 generic.go:356] "Generic (PLEG): container finished" podID="39984c71-6603-47a3-9344-7407edf3bcd2" containerID="226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be" exitCode=0 Oct 31 01:02:35.393199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.393121 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" Oct 31 01:02:35.393199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.393139 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" event={"ID":"39984c71-6603-47a3-9344-7407edf3bcd2","Type":"ContainerDied","Data":"226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be"} Oct 31 01:02:35.393199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.393190 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9" event={"ID":"39984c71-6603-47a3-9344-7407edf3bcd2","Type":"ContainerDied","Data":"a5156ad497d9df9b1bc8ec7f6de88d59de5f67e05a4c2314afeb00aff4bbc9cc"} Oct 31 01:02:35.393307 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.393222 5719 scope.go:122] "RemoveContainer" containerID="226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be" Oct 31 01:02:35.407483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.407466 5719 scope.go:122] "RemoveContainer" containerID="226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be" Oct 31 01:02:35.407768 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:02:35.407751 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be\": container with ID starting with 226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be not found: ID does not exist" containerID="226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be" Oct 31 01:02:35.407848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.407778 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be"} err="failed to get container status \"226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be\": rpc error: code = NotFound desc = could not find container \"226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be\": container with ID starting with 226c6449b659963ce0662acb35e88842573780f0929cd2f099f877987c0800be not found: ID does not exist" Oct 31 01:02:35.463800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.463664 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdfc\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-kube-api-access-4kdfc\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.463800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.463741 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39984c71-6603-47a3-9344-7407edf3bcd2-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.463800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.463760 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39984c71-6603-47a3-9344-7407edf3bcd2-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.463800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.463775 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39984c71-6603-47a3-9344-7407edf3bcd2-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.463800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.463790 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39984c71-6603-47a3-9344-7407edf3bcd2-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:35.545189 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.545133 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9"] Oct 31 01:02:35.551541 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:35.551504 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-7f5m9"] Oct 31 01:02:36.602068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:36.602034 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" path="/var/lib/kubelet/pods/39984c71-6603-47a3-9344-7407edf3bcd2/volumes" Oct 31 01:02:44.998430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:44.998399 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp"] Oct 31 01:02:44.998899 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:44.998648 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" containerID="cri-o://e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6" gracePeriod=30 Oct 31 01:02:46.576185 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:46.576153 5719 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kj5vp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:02:46.576655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:46.576249 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:02:50.254775 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.254745 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 01:02:50.321439 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321420 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-socket\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.321554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321488 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-kube-api-access-nxx4n\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-kube-api-access-nxx4n\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.321554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321515 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-credential-socket\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.321554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321539 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-istio-token\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-istio-token\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.321659 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321566 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f38d97e3-7f91-4460-b598-b659e262afb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d97e3-7f91-4460-b598-b659e262afb3-istio-podinfo\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.321731 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321681 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38d97e3-7f91-4460-b598-b659e262afb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d97e3-7f91-4460-b598-b659e262afb3-istiod-ca-cert\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.321782 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321723 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-socket" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.321782 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321732 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-certs\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.321911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321816 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-envoy\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.321911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321825 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-credential-socket" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.321911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321859 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-data\") pod \"f38d97e3-7f91-4460-b598-b659e262afb3\" (UID: \"f38d97e3-7f91-4460-b598-b659e262afb3\") " Oct 31 01:02:50.322081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.321950 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-certs" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.322157 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.322114 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38d97e3-7f91-4460-b598-b659e262afb3-istiod-ca-cert" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:02:50.322228 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.322207 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-data" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.322481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.322462 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f38d97e3-7f91-4460-b598-b659e262afb3-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.322576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.322487 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.322576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.322503 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.322576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.322515 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.322576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.322530 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.323878 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.323851 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f38d97e3-7f91-4460-b598-b659e262afb3-istio-podinfo" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:02:50.323968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.323922 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-istio-token" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:50.324336 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.324320 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-envoy" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:02:50.324393 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.324339 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-kube-api-access-nxx4n" pod "f38d97e3-7f91-4460-b598-b659e262afb3" (UID: "f38d97e3-7f91-4460-b598-b659e262afb3"). InnerVolumeSpecName "kube-api-access-nxx4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:02:50.423848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.423816 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxx4n\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-kube-api-access-nxx4n\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.423848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.423850 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f38d97e3-7f91-4460-b598-b659e262afb3-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.424063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.423865 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f38d97e3-7f91-4460-b598-b659e262afb3-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.424063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.423879 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f38d97e3-7f91-4460-b598-b659e262afb3-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:02:50.495034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.495000 5719 generic.go:356] "Generic (PLEG): container finished" podID="f38d97e3-7f91-4460-b598-b659e262afb3" containerID="e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6" exitCode=0 Oct 31 01:02:50.495235 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.495051 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" Oct 31 01:02:50.495235 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.495072 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" event={"ID":"f38d97e3-7f91-4460-b598-b659e262afb3","Type":"ContainerDied","Data":"e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6"} Oct 31 01:02:50.495235 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.495124 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp" event={"ID":"f38d97e3-7f91-4460-b598-b659e262afb3","Type":"ContainerDied","Data":"db746780c5bbdebae53967a89a02e6975aebe4980d48f97f55f5dd1c9c8e602f"} Oct 31 01:02:50.495235 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.495154 5719 scope.go:122] "RemoveContainer" containerID="e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6" Oct 31 01:02:50.505635 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.505618 5719 scope.go:122] "RemoveContainer" containerID="e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6" Oct 31 01:02:50.506040 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:02:50.506021 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6\": container with ID starting with e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6 not found: ID does not exist" containerID="e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6" Oct 31 01:02:50.506127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.506051 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6"} err="failed to get container status \"e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6\": rpc error: code = NotFound desc = could not find container \"e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6\": container with ID starting with e201376c47e3e314839d23f833ccc5e6b0f56ee7922561d30e40cd6a22da48c6 not found: ID does not exist" Oct 31 01:02:50.655501 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.655386 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp"] Oct 31 01:02:50.658644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:50.658610 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kj5vp"] Oct 31 01:02:52.604085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:02:52.604055 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" path="/var/lib/kubelet/pods/f38d97e3-7f91-4460-b598-b659e262afb3/volumes" Oct 31 01:03:03.013236 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:03:03.010568 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:03:03.240495 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:03.240465 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:03:14.450105 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450073 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk"] Oct 31 01:03:14.450553 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450537 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0ff3b0fe-841c-4c48-a0ea-12209349547a" containerName="revoke-keys" Oct 31 01:03:14.450615 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450555 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff3b0fe-841c-4c48-a0ea-12209349547a" containerName="revoke-keys" Oct 31 01:03:14.450615 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450585 5719 cpu_manager.go:414] "containerMap: removing container" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" Oct 31 01:03:14.450615 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450591 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" Oct 31 01:03:14.450731 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450618 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" Oct 31 01:03:14.450731 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450623 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" Oct 31 01:03:14.450796 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450738 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38d97e3-7f91-4460-b598-b659e262afb3" containerName="istio-proxy" Oct 31 01:03:14.450796 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450749 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff3b0fe-841c-4c48-a0ea-12209349547a" containerName="revoke-keys" Oct 31 01:03:14.450796 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.450760 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="39984c71-6603-47a3-9344-7407edf3bcd2" containerName="istio-proxy" Oct 31 01:03:14.454402 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.454382 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.510007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.509969 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:03:14.561857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.561833 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:03:14.561857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.561861 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:03:14.561857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.561880 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:03:14.564234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.564213 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk"] Oct 31 01:03:14.566458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.566433 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkrvz\" (UniqueName: \"kubernetes.io/projected/cf02772c-86d4-4419-bdeb-c4cd2d386212-kube-api-access-wkrvz\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.566570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.566500 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.566643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.566624 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.667812 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.667785 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.667948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.667868 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkrvz\" (UniqueName: \"kubernetes.io/projected/cf02772c-86d4-4419-bdeb-c4cd2d386212-kube-api-access-wkrvz\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.667948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.667909 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.668488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.668463 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.670455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.670433 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.727358 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.727293 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkrvz\" (UniqueName: \"kubernetes.io/projected/cf02772c-86d4-4419-bdeb-c4cd2d386212-kube-api-access-wkrvz\") pod \"maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:14.767210 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:14.767180 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:15.163922 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:15.163891 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk"] Oct 31 01:03:15.169208 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:03:15.169180 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf02772c_86d4_4419_bdeb_c4cd2d386212.slice/crio-640f5d07bc78d366f00bec2e8c7c3e756e006239b45369b7d0e65aa0d2b95bd2 WatchSource:0}: Error finding container 640f5d07bc78d366f00bec2e8c7c3e756e006239b45369b7d0e65aa0d2b95bd2: Status 404 returned error can't find the container with id 640f5d07bc78d366f00bec2e8c7c3e756e006239b45369b7d0e65aa0d2b95bd2 Oct 31 01:03:15.645509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:15.645472 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" event={"ID":"cf02772c-86d4-4419-bdeb-c4cd2d386212","Type":"ContainerStarted","Data":"8536865e872482bbef1d7b914b0a7fd38de67b72c1d146e806783f18db8a987d"} Oct 31 01:03:15.645933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:15.645559 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" event={"ID":"cf02772c-86d4-4419-bdeb-c4cd2d386212","Type":"ContainerStarted","Data":"640f5d07bc78d366f00bec2e8c7c3e756e006239b45369b7d0e65aa0d2b95bd2"} Oct 31 01:03:15.765580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:15.765488 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" podStartSLOduration=1.765474345 podStartE2EDuration="1.765474345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:15.763549693 +0000 UTC m=+2077.299691997" watchObservedRunningTime="2026-10-31 01:03:15.765474345 +0000 UTC m=+2077.301616649" Oct 31 01:03:16.652497 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:16.652465 5719 generic.go:356] "Generic (PLEG): container finished" podID="cf02772c-86d4-4419-bdeb-c4cd2d386212" containerID="8536865e872482bbef1d7b914b0a7fd38de67b72c1d146e806783f18db8a987d" exitCode=0 Oct 31 01:03:16.652947 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:16.652554 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" event={"ID":"cf02772c-86d4-4419-bdeb-c4cd2d386212","Type":"ContainerDied","Data":"8536865e872482bbef1d7b914b0a7fd38de67b72c1d146e806783f18db8a987d"} Oct 31 01:03:17.808393 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:17.808371 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:17.902817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:17.902790 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-openshift-ca\") pod \"cf02772c-86d4-4419-bdeb-c4cd2d386212\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " Oct 31 01:03:17.903003 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:17.902836 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf02772c-86d4-4419-bdeb-c4cd2d386212-kube-api-access-wkrvz\" (UniqueName: \"kubernetes.io/projected/cf02772c-86d4-4419-bdeb-c4cd2d386212-kube-api-access-wkrvz\") pod \"cf02772c-86d4-4419-bdeb-c4cd2d386212\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " Oct 31 01:03:17.903003 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:17.902888 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-xks-ca\") pod \"cf02772c-86d4-4419-bdeb-c4cd2d386212\" (UID: \"cf02772c-86d4-4419-bdeb-c4cd2d386212\") " Oct 31 01:03:17.903213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:17.903188 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-openshift-ca" pod "cf02772c-86d4-4419-bdeb-c4cd2d386212" (UID: "cf02772c-86d4-4419-bdeb-c4cd2d386212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:17.903365 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:17.903349 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:17.905220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:17.905198 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-xks-ca" pod "cf02772c-86d4-4419-bdeb-c4cd2d386212" (UID: "cf02772c-86d4-4419-bdeb-c4cd2d386212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:17.905309 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:17.905246 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf02772c-86d4-4419-bdeb-c4cd2d386212-kube-api-access-wkrvz" pod "cf02772c-86d4-4419-bdeb-c4cd2d386212" (UID: "cf02772c-86d4-4419-bdeb-c4cd2d386212"). InnerVolumeSpecName "kube-api-access-wkrvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:18.004629 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:18.004599 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkrvz\" (UniqueName: \"kubernetes.io/projected/cf02772c-86d4-4419-bdeb-c4cd2d386212-kube-api-access-wkrvz\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:18.004629 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:18.004625 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf02772c-86d4-4419-bdeb-c4cd2d386212-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:18.664626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:18.664595 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" event={"ID":"cf02772c-86d4-4419-bdeb-c4cd2d386212","Type":"ContainerDied","Data":"640f5d07bc78d366f00bec2e8c7c3e756e006239b45369b7d0e65aa0d2b95bd2"} Oct 31 01:03:18.664626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:18.664613 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk" Oct 31 01:03:18.664805 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:18.664636 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640f5d07bc78d366f00bec2e8c7c3e756e006239b45369b7d0e65aa0d2b95bd2" Oct 31 01:03:27.005761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.005721 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9"] Oct 31 01:03:27.006397 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.006376 5719 cpu_manager.go:414] "containerMap: removing container" podUID="cf02772c-86d4-4419-bdeb-c4cd2d386212" containerName="revoke-keys" Oct 31 01:03:27.006485 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.006399 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf02772c-86d4-4419-bdeb-c4cd2d386212" containerName="revoke-keys" Oct 31 01:03:27.006563 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.006551 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf02772c-86d4-4419-bdeb-c4cd2d386212" containerName="revoke-keys" Oct 31 01:03:27.009958 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.009937 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.065275 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.065242 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9"] Oct 31 01:03:27.104311 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.104276 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw8ht\" (UniqueName: \"kubernetes.io/projected/ffb46a09-1c57-438c-9a25-f465d1f0f79b-kube-api-access-hw8ht\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.104502 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.104435 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.104575 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.104537 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.110281 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.110259 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:03:27.110281 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.110262 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:03:27.110281 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.110264 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:03:27.110281 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.110286 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:03:27.205305 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.205265 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.205491 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.205362 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.205491 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.205412 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw8ht\" (UniqueName: \"kubernetes.io/projected/ffb46a09-1c57-438c-9a25-f465d1f0f79b-kube-api-access-hw8ht\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.206026 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.206002 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.207830 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.207812 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.265112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.265042 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw8ht\" (UniqueName: \"kubernetes.io/projected/ffb46a09-1c57-438c-9a25-f465d1f0f79b-kube-api-access-hw8ht\") pod \"maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.322854 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.322819 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:27.697562 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.697529 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9"] Oct 31 01:03:27.702529 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:03:27.702506 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb46a09_1c57_438c_9a25_f465d1f0f79b.slice/crio-4ff943e9b2501d01ed59b878ef75f944ed6dc200223a004f10d5c7e8d6be609d WatchSource:0}: Error finding container 4ff943e9b2501d01ed59b878ef75f944ed6dc200223a004f10d5c7e8d6be609d: Status 404 returned error can't find the container with id 4ff943e9b2501d01ed59b878ef75f944ed6dc200223a004f10d5c7e8d6be609d Oct 31 01:03:27.726080 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:27.726055 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" event={"ID":"ffb46a09-1c57-438c-9a25-f465d1f0f79b","Type":"ContainerStarted","Data":"4ff943e9b2501d01ed59b878ef75f944ed6dc200223a004f10d5c7e8d6be609d"} Oct 31 01:03:28.733574 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:28.733536 5719 generic.go:356] "Generic (PLEG): container finished" podID="ffb46a09-1c57-438c-9a25-f465d1f0f79b" containerID="4e45196583cf738b0f6d5dd4206b1bb90ddc7d099a3878f7b6a2d2524abe5f5e" exitCode=0 Oct 31 01:03:28.733993 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:28.733570 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" event={"ID":"ffb46a09-1c57-438c-9a25-f465d1f0f79b","Type":"ContainerDied","Data":"4e45196583cf738b0f6d5dd4206b1bb90ddc7d099a3878f7b6a2d2524abe5f5e"} Oct 31 01:03:29.886301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:29.886277 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:30.038526 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.038450 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb46a09-1c57-438c-9a25-f465d1f0f79b-kube-api-access-hw8ht\" (UniqueName: \"kubernetes.io/projected/ffb46a09-1c57-438c-9a25-f465d1f0f79b-kube-api-access-hw8ht\") pod \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " Oct 31 01:03:30.038756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.038594 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-xks-ca\") pod \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " Oct 31 01:03:30.038756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.038632 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-openshift-ca\") pod \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\" (UID: \"ffb46a09-1c57-438c-9a25-f465d1f0f79b\") " Oct 31 01:03:30.039027 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.039000 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-openshift-ca" pod "ffb46a09-1c57-438c-9a25-f465d1f0f79b" (UID: "ffb46a09-1c57-438c-9a25-f465d1f0f79b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:30.040865 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.040842 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-xks-ca" pod "ffb46a09-1c57-438c-9a25-f465d1f0f79b" (UID: "ffb46a09-1c57-438c-9a25-f465d1f0f79b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:03:30.040952 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.040914 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb46a09-1c57-438c-9a25-f465d1f0f79b-kube-api-access-hw8ht" pod "ffb46a09-1c57-438c-9a25-f465d1f0f79b" (UID: "ffb46a09-1c57-438c-9a25-f465d1f0f79b"). InnerVolumeSpecName "kube-api-access-hw8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:30.140259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.140222 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw8ht\" (UniqueName: \"kubernetes.io/projected/ffb46a09-1c57-438c-9a25-f465d1f0f79b-kube-api-access-hw8ht\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:30.140259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.140252 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:30.140259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.140261 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffb46a09-1c57-438c-9a25-f465d1f0f79b-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:03:30.746356 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.746324 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" Oct 31 01:03:30.746625 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.746357 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9" event={"ID":"ffb46a09-1c57-438c-9a25-f465d1f0f79b","Type":"ContainerDied","Data":"4ff943e9b2501d01ed59b878ef75f944ed6dc200223a004f10d5c7e8d6be609d"} Oct 31 01:03:30.746793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:30.746645 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff943e9b2501d01ed59b878ef75f944ed6dc200223a004f10d5c7e8d6be609d" Oct 31 01:03:33.009645 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:03:33.009612 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:03:40.941671 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:40.941640 5719 scope.go:122] "RemoveContainer" containerID="51afcb1100164c5a3df5de21251eefd3ecfa5e2c06fa0677bfa16d9e9d4ac7f0" Oct 31 01:03:40.953205 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:40.953181 5719 scope.go:122] "RemoveContainer" containerID="4a12a7c6ace2e353c46c38a424f8562ae8ecd5ad6cb63201cc5759709656a6a8" Oct 31 01:03:42.270809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:42.270777 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t"] Oct 31 01:03:42.318094 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:42.318068 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-eb41d3-6kn7t"] Oct 31 01:03:42.603817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:42.603791 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff3b0fe-841c-4c48-a0ea-12209349547a" path="/var/lib/kubelet/pods/0ff3b0fe-841c-4c48-a0ea-12209349547a/volumes" Oct 31 01:03:52.308066 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:52.308025 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:03:55.158603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:55.158563 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9"] Oct 31 01:03:55.206795 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:55.206759 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-cb2b8a-bd4t9"] Oct 31 01:03:56.603992 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:03:56.603957 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb46a09-1c57-438c-9a25-f465d1f0f79b" path="/var/lib/kubelet/pods/ffb46a09-1c57-438c-9a25-f465d1f0f79b/volumes" Oct 31 01:04:03.010346 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:04:03.010304 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:04:13.275580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.275540 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8"] Oct 31 01:04:13.276243 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.276224 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ffb46a09-1c57-438c-9a25-f465d1f0f79b" containerName="revoke-keys" Oct 31 01:04:13.276243 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.276243 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb46a09-1c57-438c-9a25-f465d1f0f79b" containerName="revoke-keys" Oct 31 01:04:13.276398 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.276370 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb46a09-1c57-438c-9a25-f465d1f0f79b" containerName="revoke-keys" Oct 31 01:04:13.279405 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.279387 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.329882 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.329858 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:13.329980 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.329881 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:04:13.330034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.330017 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:04:13.377905 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.377879 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:04:13.380192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.380168 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8"] Oct 31 01:04:13.407555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.407532 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvp6v\" (UniqueName: \"kubernetes.io/projected/03e133a8-2650-4e40-a7db-a2fa71e7a511-kube-api-access-qvp6v\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.407659 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.407615 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.407788 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.407767 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.508473 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.508450 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.508580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.508508 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.508621 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.508611 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvp6v\" (UniqueName: \"kubernetes.io/projected/03e133a8-2650-4e40-a7db-a2fa71e7a511-kube-api-access-qvp6v\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.509133 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.509114 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.510996 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.510977 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.567623 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.567603 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvp6v\" (UniqueName: \"kubernetes.io/projected/03e133a8-2650-4e40-a7db-a2fa71e7a511-kube-api-access-qvp6v\") pod \"maas-api-revoke-sub-b0d6b5ab-lr4t8\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.592388 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.592368 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:13.968422 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:13.968343 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8"] Oct 31 01:04:13.973745 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:04:13.973718 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e133a8_2650_4e40_a7db_a2fa71e7a511.slice/crio-6449eb93bddb850b8caade941963fe22f71114e75fa42779c8ad98e734e61113 WatchSource:0}: Error finding container 6449eb93bddb850b8caade941963fe22f71114e75fa42779c8ad98e734e61113: Status 404 returned error can't find the container with id 6449eb93bddb850b8caade941963fe22f71114e75fa42779c8ad98e734e61113 Oct 31 01:04:14.011754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:14.011730 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" event={"ID":"03e133a8-2650-4e40-a7db-a2fa71e7a511","Type":"ContainerStarted","Data":"6449eb93bddb850b8caade941963fe22f71114e75fa42779c8ad98e734e61113"} Oct 31 01:04:15.018293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:15.018257 5719 generic.go:356] "Generic (PLEG): container finished" podID="03e133a8-2650-4e40-a7db-a2fa71e7a511" containerID="4c726103c81409c56cda094ff5edc76bc1c9518cc544911fd25e7418e45951d8" exitCode=0 Oct 31 01:04:15.018724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:15.018292 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" event={"ID":"03e133a8-2650-4e40-a7db-a2fa71e7a511","Type":"ContainerDied","Data":"4c726103c81409c56cda094ff5edc76bc1c9518cc544911fd25e7418e45951d8"} Oct 31 01:04:16.166586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.166561 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:16.238487 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.238455 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-xks-ca\") pod \"03e133a8-2650-4e40-a7db-a2fa71e7a511\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " Oct 31 01:04:16.238637 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.238514 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e133a8-2650-4e40-a7db-a2fa71e7a511-kube-api-access-qvp6v\" (UniqueName: \"kubernetes.io/projected/03e133a8-2650-4e40-a7db-a2fa71e7a511-kube-api-access-qvp6v\") pod \"03e133a8-2650-4e40-a7db-a2fa71e7a511\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " Oct 31 01:04:16.238637 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.238629 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-openshift-ca\") pod \"03e133a8-2650-4e40-a7db-a2fa71e7a511\" (UID: \"03e133a8-2650-4e40-a7db-a2fa71e7a511\") " Oct 31 01:04:16.239043 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.239015 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-openshift-ca" pod "03e133a8-2650-4e40-a7db-a2fa71e7a511" (UID: "03e133a8-2650-4e40-a7db-a2fa71e7a511"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:16.239171 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.239140 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:16.240666 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.240643 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-xks-ca" pod "03e133a8-2650-4e40-a7db-a2fa71e7a511" (UID: "03e133a8-2650-4e40-a7db-a2fa71e7a511"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:16.240790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.240734 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e133a8-2650-4e40-a7db-a2fa71e7a511-kube-api-access-qvp6v" pod "03e133a8-2650-4e40-a7db-a2fa71e7a511" (UID: "03e133a8-2650-4e40-a7db-a2fa71e7a511"). InnerVolumeSpecName "kube-api-access-qvp6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:16.339752 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.339717 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e133a8-2650-4e40-a7db-a2fa71e7a511-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:16.339752 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:16.339745 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvp6v\" (UniqueName: \"kubernetes.io/projected/03e133a8-2650-4e40-a7db-a2fa71e7a511-kube-api-access-qvp6v\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:17.030684 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:17.030609 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" event={"ID":"03e133a8-2650-4e40-a7db-a2fa71e7a511","Type":"ContainerDied","Data":"6449eb93bddb850b8caade941963fe22f71114e75fa42779c8ad98e734e61113"} Oct 31 01:04:17.030684 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:17.030656 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6449eb93bddb850b8caade941963fe22f71114e75fa42779c8ad98e734e61113" Oct 31 01:04:17.030684 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:17.030622 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8" Oct 31 01:04:22.505381 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:22.505352 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:04:29.994832 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:29.994797 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx"] Oct 31 01:04:29.995299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:29.994987 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" podUID="33103902-63e1-4785-8af4-59501cb92cf4" containerName="discovery" containerID="cri-o://2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6" gracePeriod=30 Oct 31 01:04:30.270438 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.270410 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 01:04:30.385285 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.385256 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-cacerts\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-cacerts\") pod \"33103902-63e1-4785-8af4-59501cb92cf4\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " Oct 31 01:04:30.385458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.385319 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33103902-63e1-4785-8af4-59501cb92cf4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33103902-63e1-4785-8af4-59501cb92cf4-local-certs\") pod \"33103902-63e1-4785-8af4-59501cb92cf4\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " Oct 31 01:04:30.385458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.385390 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-kubeconfig\") pod \"33103902-63e1-4785-8af4-59501cb92cf4\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " Oct 31 01:04:30.385571 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.385552 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-kube-api-access-snk6d\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-kube-api-access-snk6d\") pod \"33103902-63e1-4785-8af4-59501cb92cf4\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " Oct 31 01:04:30.385626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.385588 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-dns-cert\") pod \"33103902-63e1-4785-8af4-59501cb92cf4\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " Oct 31 01:04:30.385680 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.385625 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-ca-configmap\") pod \"33103902-63e1-4785-8af4-59501cb92cf4\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " Oct 31 01:04:30.385777 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.385736 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-istio-token\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-istio-token\") pod \"33103902-63e1-4785-8af4-59501cb92cf4\" (UID: \"33103902-63e1-4785-8af4-59501cb92cf4\") " Oct 31 01:04:30.386171 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.386144 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-ca-configmap" pod "33103902-63e1-4785-8af4-59501cb92cf4" (UID: "33103902-63e1-4785-8af4-59501cb92cf4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:30.388311 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.388282 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-kubeconfig" pod "33103902-63e1-4785-8af4-59501cb92cf4" (UID: "33103902-63e1-4785-8af4-59501cb92cf4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:30.388442 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.388302 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-cacerts" pod "33103902-63e1-4785-8af4-59501cb92cf4" (UID: "33103902-63e1-4785-8af4-59501cb92cf4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:30.388511 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.388472 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33103902-63e1-4785-8af4-59501cb92cf4-local-certs" pod "33103902-63e1-4785-8af4-59501cb92cf4" (UID: "33103902-63e1-4785-8af4-59501cb92cf4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:04:30.388555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.388511 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-istio-token" pod "33103902-63e1-4785-8af4-59501cb92cf4" (UID: "33103902-63e1-4785-8af4-59501cb92cf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:30.388555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.388538 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-kube-api-access-snk6d" pod "33103902-63e1-4785-8af4-59501cb92cf4" (UID: "33103902-63e1-4785-8af4-59501cb92cf4"). InnerVolumeSpecName "kube-api-access-snk6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:30.388630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.388547 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-dns-cert" pod "33103902-63e1-4785-8af4-59501cb92cf4" (UID: "33103902-63e1-4785-8af4-59501cb92cf4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:30.487917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.487884 5719 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33103902-63e1-4785-8af4-59501cb92cf4-local-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:30.487917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.487911 5719 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-kubeconfig\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:30.487917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.487922 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snk6d\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-kube-api-access-snk6d\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:30.488165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.487931 5719 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-dns-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:30.488165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.487940 5719 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33103902-63e1-4785-8af4-59501cb92cf4-istio-csr-ca-configmap\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:30.488165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.487949 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33103902-63e1-4785-8af4-59501cb92cf4-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:30.488165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:30.487958 5719 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33103902-63e1-4785-8af4-59501cb92cf4-cacerts\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:31.117822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.117790 5719 generic.go:356] "Generic (PLEG): container finished" podID="33103902-63e1-4785-8af4-59501cb92cf4" containerID="2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6" exitCode=0 Oct 31 01:04:31.118302 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.117839 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" Oct 31 01:04:31.118302 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.117881 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" event={"ID":"33103902-63e1-4785-8af4-59501cb92cf4","Type":"ContainerDied","Data":"2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6"} Oct 31 01:04:31.118302 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.117939 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx" event={"ID":"33103902-63e1-4785-8af4-59501cb92cf4","Type":"ContainerDied","Data":"9d1f2162a784b1feafcc17a5eaf0d94445d7547714603058c1f5b66936492dc7"} Oct 31 01:04:31.118302 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.117969 5719 scope.go:122] "RemoveContainer" containerID="2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6" Oct 31 01:04:31.128133 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.128115 5719 scope.go:122] "RemoveContainer" containerID="2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6" Oct 31 01:04:31.128420 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:04:31.128398 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6\": container with ID starting with 2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6 not found: ID does not exist" containerID="2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6" Oct 31 01:04:31.128521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.128433 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6"} err="failed to get container status \"2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6\": rpc error: code = NotFound desc = could not find container \"2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6\": container with ID starting with 2ea8c7e2bb9d528ce0313d6e9acc37f6129c7fe3936af601493ca0ae137875e6 not found: ID does not exist" Oct 31 01:04:31.306441 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.306392 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx"] Oct 31 01:04:31.352324 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.352290 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5ctsx"] Oct 31 01:04:31.571038 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.571009 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z"] Oct 31 01:04:31.571492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.571479 5719 cpu_manager.go:414] "containerMap: removing container" podUID="03e133a8-2650-4e40-a7db-a2fa71e7a511" containerName="revoke-keys" Oct 31 01:04:31.571545 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.571495 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e133a8-2650-4e40-a7db-a2fa71e7a511" containerName="revoke-keys" Oct 31 01:04:31.571577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.571529 5719 cpu_manager.go:414] "containerMap: removing container" podUID="33103902-63e1-4785-8af4-59501cb92cf4" containerName="discovery" Oct 31 01:04:31.571577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.571551 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33103902-63e1-4785-8af4-59501cb92cf4" containerName="discovery" Oct 31 01:04:31.571649 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.571635 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="33103902-63e1-4785-8af4-59501cb92cf4" containerName="discovery" Oct 31 01:04:31.571714 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.571651 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e133a8-2650-4e40-a7db-a2fa71e7a511" containerName="revoke-keys" Oct 31 01:04:31.576321 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.576304 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.627425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.627397 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:31.627529 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.627448 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:04:31.627997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.627982 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:04:31.628124 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.628111 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:04:31.636346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.636326 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z"] Oct 31 01:04:31.701555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.701527 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.701665 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.701580 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.701746 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.701681 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22cr\" (UniqueName: \"kubernetes.io/projected/97054a4a-835c-4bf9-a96b-f8f29726bac5-kube-api-access-h22cr\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.803141 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.803108 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.803246 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.803165 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.803291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.803276 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22cr\" (UniqueName: \"kubernetes.io/projected/97054a4a-835c-4bf9-a96b-f8f29726bac5-kube-api-access-h22cr\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.803888 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.803824 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.805498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.805479 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.873510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.873464 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22cr\" (UniqueName: \"kubernetes.io/projected/97054a4a-835c-4bf9-a96b-f8f29726bac5-kube-api-access-h22cr\") pod \"maas-api-revoke-sub-10f688fd-l298z\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:31.889271 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:31.889253 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:32.274389 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:32.274320 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z"] Oct 31 01:04:32.279757 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:04:32.279730 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97054a4a_835c_4bf9_a96b_f8f29726bac5.slice/crio-ed3346a5991d84bf3aef61556d482bbf5afcf0523ee5e400cabca5ef296cb5ad WatchSource:0}: Error finding container ed3346a5991d84bf3aef61556d482bbf5afcf0523ee5e400cabca5ef296cb5ad: Status 404 returned error can't find the container with id ed3346a5991d84bf3aef61556d482bbf5afcf0523ee5e400cabca5ef296cb5ad Oct 31 01:04:32.603195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:32.603168 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33103902-63e1-4785-8af4-59501cb92cf4" path="/var/lib/kubelet/pods/33103902-63e1-4785-8af4-59501cb92cf4/volumes" Oct 31 01:04:33.009334 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:04:33.009259 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:04:33.131939 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:33.131907 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" event={"ID":"97054a4a-835c-4bf9-a96b-f8f29726bac5","Type":"ContainerStarted","Data":"761037df0e59824e30d58329d8201cce484fd5666cb43161df416360bda648de"} Oct 31 01:04:33.132110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:33.131963 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" event={"ID":"97054a4a-835c-4bf9-a96b-f8f29726bac5","Type":"ContainerStarted","Data":"ed3346a5991d84bf3aef61556d482bbf5afcf0523ee5e400cabca5ef296cb5ad"} Oct 31 01:04:34.139404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:34.139363 5719 generic.go:356] "Generic (PLEG): container finished" podID="97054a4a-835c-4bf9-a96b-f8f29726bac5" containerID="761037df0e59824e30d58329d8201cce484fd5666cb43161df416360bda648de" exitCode=0 Oct 31 01:04:34.139822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:34.139403 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" event={"ID":"97054a4a-835c-4bf9-a96b-f8f29726bac5","Type":"ContainerDied","Data":"761037df0e59824e30d58329d8201cce484fd5666cb43161df416360bda648de"} Oct 31 01:04:35.278013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.277987 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:35.438822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.438734 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-xks-ca\") pod \"97054a4a-835c-4bf9-a96b-f8f29726bac5\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " Oct 31 01:04:35.438973 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.438853 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-openshift-ca\") pod \"97054a4a-835c-4bf9-a96b-f8f29726bac5\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " Oct 31 01:04:35.438973 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.438882 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97054a4a-835c-4bf9-a96b-f8f29726bac5-kube-api-access-h22cr\" (UniqueName: \"kubernetes.io/projected/97054a4a-835c-4bf9-a96b-f8f29726bac5-kube-api-access-h22cr\") pod \"97054a4a-835c-4bf9-a96b-f8f29726bac5\" (UID: \"97054a4a-835c-4bf9-a96b-f8f29726bac5\") " Oct 31 01:04:35.439227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.439207 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-openshift-ca" pod "97054a4a-835c-4bf9-a96b-f8f29726bac5" (UID: "97054a4a-835c-4bf9-a96b-f8f29726bac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:35.439375 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.439360 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:35.440929 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.440899 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-xks-ca" pod "97054a4a-835c-4bf9-a96b-f8f29726bac5" (UID: "97054a4a-835c-4bf9-a96b-f8f29726bac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:35.441032 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.441012 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97054a4a-835c-4bf9-a96b-f8f29726bac5-kube-api-access-h22cr" pod "97054a4a-835c-4bf9-a96b-f8f29726bac5" (UID: "97054a4a-835c-4bf9-a96b-f8f29726bac5"). InnerVolumeSpecName "kube-api-access-h22cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:35.540296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.540262 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97054a4a-835c-4bf9-a96b-f8f29726bac5-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:35.540296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:35.540294 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22cr\" (UniqueName: \"kubernetes.io/projected/97054a4a-835c-4bf9-a96b-f8f29726bac5-kube-api-access-h22cr\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:36.154519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:36.154488 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" Oct 31 01:04:36.154664 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:36.154525 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z" event={"ID":"97054a4a-835c-4bf9-a96b-f8f29726bac5","Type":"ContainerDied","Data":"ed3346a5991d84bf3aef61556d482bbf5afcf0523ee5e400cabca5ef296cb5ad"} Oct 31 01:04:36.154664 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:36.154570 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed3346a5991d84bf3aef61556d482bbf5afcf0523ee5e400cabca5ef296cb5ad" Oct 31 01:04:48.079370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.079291 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr"] Oct 31 01:04:48.079800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.079786 5719 cpu_manager.go:414] "containerMap: removing container" podUID="97054a4a-835c-4bf9-a96b-f8f29726bac5" containerName="revoke-keys" Oct 31 01:04:48.079849 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.079801 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97054a4a-835c-4bf9-a96b-f8f29726bac5" containerName="revoke-keys" Oct 31 01:04:48.079904 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.079894 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="97054a4a-835c-4bf9-a96b-f8f29726bac5" containerName="revoke-keys" Oct 31 01:04:48.084549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.084528 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.138281 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.138238 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:04:48.138457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.138426 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:04:48.138545 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.138477 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:04:48.140505 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.140478 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr"] Oct 31 01:04:48.176874 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.176846 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.177036 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.176890 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54hjv\" (UniqueName: \"kubernetes.io/projected/b29d6c03-12ba-4687-b4c7-755c60313cac-kube-api-access-54hjv\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.177036 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.176977 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.184874 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.184853 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:04:48.277670 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.277633 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.277873 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.277687 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54hjv\" (UniqueName: \"kubernetes.io/projected/b29d6c03-12ba-4687-b4c7-755c60313cac-kube-api-access-54hjv\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.277873 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.277779 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.278455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.278396 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.280177 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.280155 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.337318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.337246 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54hjv\" (UniqueName: \"kubernetes.io/projected/b29d6c03-12ba-4687-b4c7-755c60313cac-kube-api-access-54hjv\") pod \"maas-api-revoke-sub-ef7fbcf9-7qmdr\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.397526 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.397492 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:48.784953 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:48.784877 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr"] Oct 31 01:04:48.790955 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:04:48.790930 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29d6c03_12ba_4687_b4c7_755c60313cac.slice/crio-f8f3477586e335e5279cffce2f2129db9a083e2eb3b4659084147e9495e1a554 WatchSource:0}: Error finding container f8f3477586e335e5279cffce2f2129db9a083e2eb3b4659084147e9495e1a554: Status 404 returned error can't find the container with id f8f3477586e335e5279cffce2f2129db9a083e2eb3b4659084147e9495e1a554 Oct 31 01:04:49.227411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:49.227373 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" event={"ID":"b29d6c03-12ba-4687-b4c7-755c60313cac","Type":"ContainerStarted","Data":"38880015a7ad5ab9c867486e79d9035c2129def746044a0efabe090910210601"} Oct 31 01:04:49.227822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:49.227422 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" event={"ID":"b29d6c03-12ba-4687-b4c7-755c60313cac","Type":"ContainerStarted","Data":"f8f3477586e335e5279cffce2f2129db9a083e2eb3b4659084147e9495e1a554"} Oct 31 01:04:49.343275 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:49.343223 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" podStartSLOduration=1.3432094860000001 podStartE2EDuration="1.343209486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:49.342964569 +0000 UTC m=+2170.879106873" watchObservedRunningTime="2026-10-31 01:04:49.343209486 +0000 UTC m=+2170.879352161" Oct 31 01:04:50.234787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:50.234755 5719 generic.go:356] "Generic (PLEG): container finished" podID="b29d6c03-12ba-4687-b4c7-755c60313cac" containerID="38880015a7ad5ab9c867486e79d9035c2129def746044a0efabe090910210601" exitCode=0 Oct 31 01:04:50.235209 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:50.234853 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" event={"ID":"b29d6c03-12ba-4687-b4c7-755c60313cac","Type":"ContainerDied","Data":"38880015a7ad5ab9c867486e79d9035c2129def746044a0efabe090910210601"} Oct 31 01:04:51.387477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.387452 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:51.410738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.410687 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-openshift-ca\") pod \"b29d6c03-12ba-4687-b4c7-755c60313cac\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " Oct 31 01:04:51.410862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.410795 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-xks-ca\") pod \"b29d6c03-12ba-4687-b4c7-755c60313cac\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " Oct 31 01:04:51.410862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.410840 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29d6c03-12ba-4687-b4c7-755c60313cac-kube-api-access-54hjv\" (UniqueName: \"kubernetes.io/projected/b29d6c03-12ba-4687-b4c7-755c60313cac-kube-api-access-54hjv\") pod \"b29d6c03-12ba-4687-b4c7-755c60313cac\" (UID: \"b29d6c03-12ba-4687-b4c7-755c60313cac\") " Oct 31 01:04:51.411095 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.411065 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-openshift-ca" pod "b29d6c03-12ba-4687-b4c7-755c60313cac" (UID: "b29d6c03-12ba-4687-b4c7-755c60313cac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:04:51.411313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.411298 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:51.412982 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.412957 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-xks-ca" pod "b29d6c03-12ba-4687-b4c7-755c60313cac" (UID: "b29d6c03-12ba-4687-b4c7-755c60313cac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:51.412982 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.412965 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29d6c03-12ba-4687-b4c7-755c60313cac-kube-api-access-54hjv" pod "b29d6c03-12ba-4687-b4c7-755c60313cac" (UID: "b29d6c03-12ba-4687-b4c7-755c60313cac"). InnerVolumeSpecName "kube-api-access-54hjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:51.512346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.512270 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29d6c03-12ba-4687-b4c7-755c60313cac-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:51.512346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:51.512300 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54hjv\" (UniqueName: \"kubernetes.io/projected/b29d6c03-12ba-4687-b4c7-755c60313cac-kube-api-access-54hjv\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:04:52.248660 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:52.248621 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" Oct 31 01:04:52.248855 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:52.248618 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr" event={"ID":"b29d6c03-12ba-4687-b4c7-755c60313cac","Type":"ContainerDied","Data":"f8f3477586e335e5279cffce2f2129db9a083e2eb3b4659084147e9495e1a554"} Oct 31 01:04:52.248855 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:52.248766 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8f3477586e335e5279cffce2f2129db9a083e2eb3b4659084147e9495e1a554" Oct 31 01:04:52.432769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:52.432727 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk"] Oct 31 01:04:52.451849 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:52.451821 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eee7f1-hmchk"] Oct 31 01:04:52.601968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:04:52.601941 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf02772c-86d4-4419-bdeb-c4cd2d386212" path="/var/lib/kubelet/pods/cf02772c-86d4-4419-bdeb-c4cd2d386212/volumes" Oct 31 01:05:01.478034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.478004 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2"] Oct 31 01:05:01.478498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.478465 5719 cpu_manager.go:414] "containerMap: removing container" podUID="b29d6c03-12ba-4687-b4c7-755c60313cac" containerName="revoke-keys" Oct 31 01:05:01.478498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.478479 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29d6c03-12ba-4687-b4c7-755c60313cac" containerName="revoke-keys" Oct 31 01:05:01.478608 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.478597 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29d6c03-12ba-4687-b4c7-755c60313cac" containerName="revoke-keys" Oct 31 01:05:01.481757 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.481741 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.536302 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.536272 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:01.583038 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.583015 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:05:01.597968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.597927 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2"] Oct 31 01:05:01.606507 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.606486 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:01.606792 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.606768 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:05:01.620202 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.620166 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6c5\" (UniqueName: \"kubernetes.io/projected/13840698-87de-4af9-882f-4bf6026ba9da-kube-api-access-7g6c5\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.620304 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.620241 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.620411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.620388 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.721472 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.721444 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6c5\" (UniqueName: \"kubernetes.io/projected/13840698-87de-4af9-882f-4bf6026ba9da-kube-api-access-7g6c5\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.721603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.721491 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.721603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.721530 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.722146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.722115 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.723989 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.723961 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.785817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.785748 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6c5\" (UniqueName: \"kubernetes.io/projected/13840698-87de-4af9-882f-4bf6026ba9da-kube-api-access-7g6c5\") pod \"maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:01.794342 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:01.794319 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:02.187883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:02.187850 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2"] Oct 31 01:05:02.193315 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:05:02.193274 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13840698_87de_4af9_882f_4bf6026ba9da.slice/crio-4d5d20f9bf5618b62937431878f2e571644345afc2e436261b4ec5a722d0b686 WatchSource:0}: Error finding container 4d5d20f9bf5618b62937431878f2e571644345afc2e436261b4ec5a722d0b686: Status 404 returned error can't find the container with id 4d5d20f9bf5618b62937431878f2e571644345afc2e436261b4ec5a722d0b686 Oct 31 01:05:02.308072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:02.308045 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" event={"ID":"13840698-87de-4af9-882f-4bf6026ba9da","Type":"ContainerStarted","Data":"4d5d20f9bf5618b62937431878f2e571644345afc2e436261b4ec5a722d0b686"} Oct 31 01:05:03.011966 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:05:03.011922 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:05:03.315577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:03.315504 5719 generic.go:356] "Generic (PLEG): container finished" podID="13840698-87de-4af9-882f-4bf6026ba9da" containerID="73de2777b92366a8d87d81986c6472f7d04daf7788c40d886a82f2c1f3504596" exitCode=0 Oct 31 01:05:03.315751 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:03.315586 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" event={"ID":"13840698-87de-4af9-882f-4bf6026ba9da","Type":"ContainerDied","Data":"73de2777b92366a8d87d81986c6472f7d04daf7788c40d886a82f2c1f3504596"} Oct 31 01:05:04.315528 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.315483 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq"] Oct 31 01:05:04.319553 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.319532 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.386136 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.385684 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq"] Oct 31 01:05:04.449246 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.449219 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghztb\" (UniqueName: \"kubernetes.io/projected/5a141e3c-e314-4d2f-a107-ffcaec0b1230-kube-api-access-ghztb\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.449392 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.449356 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.449459 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.449426 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.467456 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.467436 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:04.549981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.549951 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-xks-ca\") pod \"13840698-87de-4af9-882f-4bf6026ba9da\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " Oct 31 01:05:04.550075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.549994 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-openshift-ca\") pod \"13840698-87de-4af9-882f-4bf6026ba9da\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " Oct 31 01:05:04.550146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.550111 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13840698-87de-4af9-882f-4bf6026ba9da-kube-api-access-7g6c5\" (UniqueName: \"kubernetes.io/projected/13840698-87de-4af9-882f-4bf6026ba9da-kube-api-access-7g6c5\") pod \"13840698-87de-4af9-882f-4bf6026ba9da\" (UID: \"13840698-87de-4af9-882f-4bf6026ba9da\") " Oct 31 01:05:04.550265 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.550249 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghztb\" (UniqueName: \"kubernetes.io/projected/5a141e3c-e314-4d2f-a107-ffcaec0b1230-kube-api-access-ghztb\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.550347 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.550331 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.550402 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.550360 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-openshift-ca" pod "13840698-87de-4af9-882f-4bf6026ba9da" (UID: "13840698-87de-4af9-882f-4bf6026ba9da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:04.550402 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.550384 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.550489 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.550445 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:04.551354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.551330 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.552174 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.552152 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-xks-ca" pod "13840698-87de-4af9-882f-4bf6026ba9da" (UID: "13840698-87de-4af9-882f-4bf6026ba9da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:04.552354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.552331 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13840698-87de-4af9-882f-4bf6026ba9da-kube-api-access-7g6c5" pod "13840698-87de-4af9-882f-4bf6026ba9da" (UID: "13840698-87de-4af9-882f-4bf6026ba9da"). InnerVolumeSpecName "kube-api-access-7g6c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:04.552813 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.552795 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.610740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.610675 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghztb\" (UniqueName: \"kubernetes.io/projected/5a141e3c-e314-4d2f-a107-ffcaec0b1230-kube-api-access-ghztb\") pod \"maas-api-revoke-sub-69b3f100-ztmkq\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.633741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.633686 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:04.651156 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.651133 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13840698-87de-4af9-882f-4bf6026ba9da-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:04.651238 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:04.651159 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g6c5\" (UniqueName: \"kubernetes.io/projected/13840698-87de-4af9-882f-4bf6026ba9da-kube-api-access-7g6c5\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:05.009145 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:05.009078 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq"] Oct 31 01:05:05.013782 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:05:05.013748 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a141e3c_e314_4d2f_a107_ffcaec0b1230.slice/crio-55c512d611647385bc7eb84db83c87945649a3c80d22c08b80b5304dd351f094 WatchSource:0}: Error finding container 55c512d611647385bc7eb84db83c87945649a3c80d22c08b80b5304dd351f094: Status 404 returned error can't find the container with id 55c512d611647385bc7eb84db83c87945649a3c80d22c08b80b5304dd351f094 Oct 31 01:05:05.049658 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:05.049636 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:05:05.328386 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:05.328357 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" event={"ID":"5a141e3c-e314-4d2f-a107-ffcaec0b1230","Type":"ContainerStarted","Data":"bb1e83eb03d230ddae5e753b1dca2144cce71fa5fce5878c0eec0a3aedaffba9"} Oct 31 01:05:05.328807 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:05.328415 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" event={"ID":"5a141e3c-e314-4d2f-a107-ffcaec0b1230","Type":"ContainerStarted","Data":"55c512d611647385bc7eb84db83c87945649a3c80d22c08b80b5304dd351f094"} Oct 31 01:05:05.329756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:05.329738 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" event={"ID":"13840698-87de-4af9-882f-4bf6026ba9da","Type":"ContainerDied","Data":"4d5d20f9bf5618b62937431878f2e571644345afc2e436261b4ec5a722d0b686"} Oct 31 01:05:05.329808 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:05.329771 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d5d20f9bf5618b62937431878f2e571644345afc2e436261b4ec5a722d0b686" Oct 31 01:05:05.329808 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:05.329775 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2" Oct 31 01:05:05.452565 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:05.452512 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" podStartSLOduration=1.452497546 podStartE2EDuration="1.452497546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:05.450439739 +0000 UTC m=+2186.986582045" watchObservedRunningTime="2026-10-31 01:05:05.452497546 +0000 UTC m=+2186.988639897" Oct 31 01:05:06.336480 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:06.336451 5719 generic.go:356] "Generic (PLEG): container finished" podID="5a141e3c-e314-4d2f-a107-ffcaec0b1230" containerID="bb1e83eb03d230ddae5e753b1dca2144cce71fa5fce5878c0eec0a3aedaffba9" exitCode=0 Oct 31 01:05:06.336884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:06.336544 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" event={"ID":"5a141e3c-e314-4d2f-a107-ffcaec0b1230","Type":"ContainerDied","Data":"bb1e83eb03d230ddae5e753b1dca2144cce71fa5fce5878c0eec0a3aedaffba9"} Oct 31 01:05:07.491468 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.491434 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:07.583938 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.583912 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a141e3c-e314-4d2f-a107-ffcaec0b1230-kube-api-access-ghztb\" (UniqueName: \"kubernetes.io/projected/5a141e3c-e314-4d2f-a107-ffcaec0b1230-kube-api-access-ghztb\") pod \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " Oct 31 01:05:07.584122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.583951 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-openshift-ca\") pod \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " Oct 31 01:05:07.584122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.584093 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-xks-ca\") pod \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\" (UID: \"5a141e3c-e314-4d2f-a107-ffcaec0b1230\") " Oct 31 01:05:07.584398 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.584376 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-openshift-ca" pod "5a141e3c-e314-4d2f-a107-ffcaec0b1230" (UID: "5a141e3c-e314-4d2f-a107-ffcaec0b1230"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:07.584563 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.584547 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:07.586225 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.586198 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a141e3c-e314-4d2f-a107-ffcaec0b1230-kube-api-access-ghztb" pod "5a141e3c-e314-4d2f-a107-ffcaec0b1230" (UID: "5a141e3c-e314-4d2f-a107-ffcaec0b1230"). InnerVolumeSpecName "kube-api-access-ghztb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:07.586580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.586559 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-xks-ca" pod "5a141e3c-e314-4d2f-a107-ffcaec0b1230" (UID: "5a141e3c-e314-4d2f-a107-ffcaec0b1230"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:07.686079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.686051 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghztb\" (UniqueName: \"kubernetes.io/projected/5a141e3c-e314-4d2f-a107-ffcaec0b1230-kube-api-access-ghztb\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:07.686079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:07.686077 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a141e3c-e314-4d2f-a107-ffcaec0b1230-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:08.350506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:08.350474 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" Oct 31 01:05:08.350733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:08.350486 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq" event={"ID":"5a141e3c-e314-4d2f-a107-ffcaec0b1230","Type":"ContainerDied","Data":"55c512d611647385bc7eb84db83c87945649a3c80d22c08b80b5304dd351f094"} Oct 31 01:05:08.350733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:08.350622 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c512d611647385bc7eb84db83c87945649a3c80d22c08b80b5304dd351f094" Oct 31 01:05:12.920831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:12.920797 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm"] Oct 31 01:05:13.026523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:13.026492 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6"] Oct 31 01:05:17.921677 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:17.921630 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" podUID="4b3465ad-042f-4aba-99f9-2286ba50b1ef" containerName="payload-processing" containerID="cri-o://42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff" gracePeriod=25 Oct 31 01:05:18.027514 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.027478 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" podUID="f5fdbb1b-6682-4bfe-8f46-af5a41fafa36" containerName="payload-pre-processing" containerID="cri-o://26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc" gracePeriod=25 Oct 31 01:05:18.190318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.190258 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:05:18.290224 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.290200 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:05:18.305492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.305473 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3465ad-042f-4aba-99f9-2286ba50b1ef-kube-api-access-8htr6\" (UniqueName: \"kubernetes.io/projected/4b3465ad-042f-4aba-99f9-2286ba50b1ef-kube-api-access-8htr6\") pod \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\" (UID: \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\") " Oct 31 01:05:18.305588 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.305509 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3465ad-042f-4aba-99f9-2286ba50b1ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3465ad-042f-4aba-99f9-2286ba50b1ef-plugins-config-volume\") pod \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\" (UID: \"4b3465ad-042f-4aba-99f9-2286ba50b1ef\") " Oct 31 01:05:18.305947 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.305928 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3465ad-042f-4aba-99f9-2286ba50b1ef-plugins-config-volume" pod "4b3465ad-042f-4aba-99f9-2286ba50b1ef" (UID: "4b3465ad-042f-4aba-99f9-2286ba50b1ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:18.307571 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.307545 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3465ad-042f-4aba-99f9-2286ba50b1ef-kube-api-access-8htr6" pod "4b3465ad-042f-4aba-99f9-2286ba50b1ef" (UID: "4b3465ad-042f-4aba-99f9-2286ba50b1ef"). InnerVolumeSpecName "kube-api-access-8htr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:18.406120 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.406093 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-plugins-config-volume\") pod \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\" (UID: \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\") " Oct 31 01:05:18.406264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.406218 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-kube-api-access-44sht\" (UniqueName: \"kubernetes.io/projected/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-kube-api-access-44sht\") pod \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\" (UID: \"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36\") " Oct 31 01:05:18.406473 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.406439 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-plugins-config-volume" pod "f5fdbb1b-6682-4bfe-8f46-af5a41fafa36" (UID: "f5fdbb1b-6682-4bfe-8f46-af5a41fafa36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:18.406636 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.406621 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:18.406713 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.406639 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8htr6\" (UniqueName: \"kubernetes.io/projected/4b3465ad-042f-4aba-99f9-2286ba50b1ef-kube-api-access-8htr6\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:18.406713 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.406648 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b3465ad-042f-4aba-99f9-2286ba50b1ef-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:18.408840 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.408818 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-kube-api-access-44sht" pod "f5fdbb1b-6682-4bfe-8f46-af5a41fafa36" (UID: "f5fdbb1b-6682-4bfe-8f46-af5a41fafa36"). InnerVolumeSpecName "kube-api-access-44sht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:18.411789 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.411767 5719 generic.go:356] "Generic (PLEG): container finished" podID="f5fdbb1b-6682-4bfe-8f46-af5a41fafa36" containerID="26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc" exitCode=0 Oct 31 01:05:18.411884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.411809 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" Oct 31 01:05:18.411884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.411848 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" event={"ID":"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36","Type":"ContainerDied","Data":"26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc"} Oct 31 01:05:18.412005 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.411918 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6" event={"ID":"f5fdbb1b-6682-4bfe-8f46-af5a41fafa36","Type":"ContainerDied","Data":"6ecc8b9855da41e8aba1a6d2ca35fd668617840a5084ee84a3225b22241a5777"} Oct 31 01:05:18.412005 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.411933 5719 scope.go:122] "RemoveContainer" containerID="26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc" Oct 31 01:05:18.413786 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.413759 5719 generic.go:356] "Generic (PLEG): container finished" podID="4b3465ad-042f-4aba-99f9-2286ba50b1ef" containerID="42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff" exitCode=0 Oct 31 01:05:18.413885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.413823 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" Oct 31 01:05:18.413885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.413829 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" event={"ID":"4b3465ad-042f-4aba-99f9-2286ba50b1ef","Type":"ContainerDied","Data":"42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff"} Oct 31 01:05:18.413885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.413874 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm" event={"ID":"4b3465ad-042f-4aba-99f9-2286ba50b1ef","Type":"ContainerDied","Data":"1ed151bce25988e957f408b4a297ce60e80199d71d941eea54027095fee7d54f"} Oct 31 01:05:18.421822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.421804 5719 scope.go:122] "RemoveContainer" containerID="26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc" Oct 31 01:05:18.422058 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:05:18.422039 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc\": container with ID starting with 26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc not found: ID does not exist" containerID="26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc" Oct 31 01:05:18.422199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.422064 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc"} err="failed to get container status \"26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc\": rpc error: code = NotFound desc = could not find container \"26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc\": container with ID starting with 26093567a7f082066ae0dd81b89d2dd85959d86ddc5e190db536ec032b26d9cc not found: ID does not exist" Oct 31 01:05:18.422199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.422085 5719 scope.go:122] "RemoveContainer" containerID="42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff" Oct 31 01:05:18.431192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.431175 5719 scope.go:122] "RemoveContainer" containerID="42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff" Oct 31 01:05:18.431536 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:05:18.431513 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff\": container with ID starting with 42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff not found: ID does not exist" containerID="42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff" Oct 31 01:05:18.431616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.431546 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff"} err="failed to get container status \"42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff\": rpc error: code = NotFound desc = could not find container \"42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff\": container with ID starting with 42f6c41986f320e89e99d8566ba680ef35fb6908f38e819dd64f27803536d7ff not found: ID does not exist" Oct 31 01:05:18.507285 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.507199 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44sht\" (UniqueName: \"kubernetes.io/projected/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36-kube-api-access-44sht\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:18.543613 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.543575 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6"] Oct 31 01:05:18.550778 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.550754 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31ae40-bc6fdccdc-rrbs6"] Oct 31 01:05:18.602913 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.602885 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fdbb1b-6682-4bfe-8f46-af5a41fafa36" path="/var/lib/kubelet/pods/f5fdbb1b-6682-4bfe-8f46-af5a41fafa36/volumes" Oct 31 01:05:18.660620 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.660585 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm"] Oct 31 01:05:18.667542 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:18.667517 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31ae40-9cd8c58b9-6xlhm"] Oct 31 01:05:20.602724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:20.602665 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3465ad-042f-4aba-99f9-2286ba50b1ef" path="/var/lib/kubelet/pods/4b3465ad-042f-4aba-99f9-2286ba50b1ef/volumes" Oct 31 01:05:21.729283 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.729244 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6"] Oct 31 01:05:21.729917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.729897 5719 cpu_manager.go:414] "containerMap: removing container" podUID="13840698-87de-4af9-882f-4bf6026ba9da" containerName="revoke-keys" Oct 31 01:05:21.730010 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.729917 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13840698-87de-4af9-882f-4bf6026ba9da" containerName="revoke-keys" Oct 31 01:05:21.730090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730031 5719 cpu_manager.go:414] "containerMap: removing container" podUID="4b3465ad-042f-4aba-99f9-2286ba50b1ef" containerName="payload-processing" Oct 31 01:05:21.730090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730041 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3465ad-042f-4aba-99f9-2286ba50b1ef" containerName="payload-processing" Oct 31 01:05:21.730090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730063 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5a141e3c-e314-4d2f-a107-ffcaec0b1230" containerName="revoke-keys" Oct 31 01:05:21.730090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730070 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a141e3c-e314-4d2f-a107-ffcaec0b1230" containerName="revoke-keys" Oct 31 01:05:21.730294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730087 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f5fdbb1b-6682-4bfe-8f46-af5a41fafa36" containerName="payload-pre-processing" Oct 31 01:05:21.730294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730118 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fdbb1b-6682-4bfe-8f46-af5a41fafa36" containerName="payload-pre-processing" Oct 31 01:05:21.730294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730248 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fdbb1b-6682-4bfe-8f46-af5a41fafa36" containerName="payload-pre-processing" Oct 31 01:05:21.730294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730266 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="13840698-87de-4af9-882f-4bf6026ba9da" containerName="revoke-keys" Oct 31 01:05:21.730294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730283 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3465ad-042f-4aba-99f9-2286ba50b1ef" containerName="payload-processing" Oct 31 01:05:21.730294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.730294 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a141e3c-e314-4d2f-a107-ffcaec0b1230" containerName="revoke-keys" Oct 31 01:05:21.735091 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.735069 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:21.792204 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.792175 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6"] Oct 31 01:05:21.836364 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.836338 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86272\" (UniqueName: \"kubernetes.io/projected/70762982-ee52-4b72-a7cd-8debe6b74455-kube-api-access-86272\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:21.836477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.836387 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:21.836549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.836528 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:21.836613 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.836598 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:21.836718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.836674 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:05:21.836773 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.836678 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:21.842254 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.842241 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:05:21.937662 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.937640 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86272\" (UniqueName: \"kubernetes.io/projected/70762982-ee52-4b72-a7cd-8debe6b74455-kube-api-access-86272\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:21.937817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.937681 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:21.937817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.937757 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:21.938300 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.938283 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:21.940065 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:21.940045 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:22.003636 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:22.003586 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86272\" (UniqueName: \"kubernetes.io/projected/70762982-ee52-4b72-a7cd-8debe6b74455-kube-api-access-86272\") pod \"maas-api-revoke-sub-a0575d4f-swfh6\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:22.048018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:22.047995 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:22.423811 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:22.423761 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6"] Oct 31 01:05:22.426777 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:05:22.426748 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70762982_ee52_4b72_a7cd_8debe6b74455.slice/crio-319cf20a8440600f29924274dde396c77cb0f8e648388d0bff5a2333d32729be WatchSource:0}: Error finding container 319cf20a8440600f29924274dde396c77cb0f8e648388d0bff5a2333d32729be: Status 404 returned error can't find the container with id 319cf20a8440600f29924274dde396c77cb0f8e648388d0bff5a2333d32729be Oct 31 01:05:22.441179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:22.441095 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" event={"ID":"70762982-ee52-4b72-a7cd-8debe6b74455","Type":"ContainerStarted","Data":"319cf20a8440600f29924274dde396c77cb0f8e648388d0bff5a2333d32729be"} Oct 31 01:05:23.449542 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:23.449515 5719 generic.go:356] "Generic (PLEG): container finished" podID="70762982-ee52-4b72-a7cd-8debe6b74455" containerID="4ee6f8f4168d0dc045f09a6993ce0da2accf930daf94deb8ed0c2c8a5998d4ca" exitCode=0 Oct 31 01:05:23.449876 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:23.449616 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" event={"ID":"70762982-ee52-4b72-a7cd-8debe6b74455","Type":"ContainerDied","Data":"4ee6f8f4168d0dc045f09a6993ce0da2accf930daf94deb8ed0c2c8a5998d4ca"} Oct 31 01:05:24.618430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.618406 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:24.665621 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.665589 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70762982-ee52-4b72-a7cd-8debe6b74455-kube-api-access-86272\" (UniqueName: \"kubernetes.io/projected/70762982-ee52-4b72-a7cd-8debe6b74455-kube-api-access-86272\") pod \"70762982-ee52-4b72-a7cd-8debe6b74455\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " Oct 31 01:05:24.665808 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.665676 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-xks-ca\") pod \"70762982-ee52-4b72-a7cd-8debe6b74455\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " Oct 31 01:05:24.665808 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.665753 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-openshift-ca\") pod \"70762982-ee52-4b72-a7cd-8debe6b74455\" (UID: \"70762982-ee52-4b72-a7cd-8debe6b74455\") " Oct 31 01:05:24.666123 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.666092 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-openshift-ca" pod "70762982-ee52-4b72-a7cd-8debe6b74455" (UID: "70762982-ee52-4b72-a7cd-8debe6b74455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:24.666250 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.666211 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:24.667868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.667842 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70762982-ee52-4b72-a7cd-8debe6b74455-kube-api-access-86272" pod "70762982-ee52-4b72-a7cd-8debe6b74455" (UID: "70762982-ee52-4b72-a7cd-8debe6b74455"). InnerVolumeSpecName "kube-api-access-86272". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:24.667868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.667860 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-xks-ca" pod "70762982-ee52-4b72-a7cd-8debe6b74455" (UID: "70762982-ee52-4b72-a7cd-8debe6b74455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:24.766843 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.766769 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86272\" (UniqueName: \"kubernetes.io/projected/70762982-ee52-4b72-a7cd-8debe6b74455-kube-api-access-86272\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:24.766843 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:24.766794 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70762982-ee52-4b72-a7cd-8debe6b74455-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:25.462007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:25.461982 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" Oct 31 01:05:25.462154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:25.461982 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6" event={"ID":"70762982-ee52-4b72-a7cd-8debe6b74455","Type":"ContainerDied","Data":"319cf20a8440600f29924274dde396c77cb0f8e648388d0bff5a2333d32729be"} Oct 31 01:05:25.462154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:25.462093 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="319cf20a8440600f29924274dde396c77cb0f8e648388d0bff5a2333d32729be" Oct 31 01:05:30.024913 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.024883 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r"] Oct 31 01:05:30.025347 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.025334 5719 cpu_manager.go:414] "containerMap: removing container" podUID="70762982-ee52-4b72-a7cd-8debe6b74455" containerName="revoke-keys" Oct 31 01:05:30.025411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.025348 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70762982-ee52-4b72-a7cd-8debe6b74455" containerName="revoke-keys" Oct 31 01:05:30.025455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.025436 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="70762982-ee52-4b72-a7cd-8debe6b74455" containerName="revoke-keys" Oct 31 01:05:30.028536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.028519 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.084215 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.084187 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b63c24-openshift-default-dockercfg-58m59\"" Oct 31 01:05:30.121193 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121157 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.121356 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121199 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.121356 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121218 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwld\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-kube-api-access-mgwld\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.121356 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121306 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.121356 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121340 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.121582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121395 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.121582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121418 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d75beef-0864-46f3-9fab-1cc248c196a1-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.121582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121477 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.121582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.121537 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.134163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.134134 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r"] Oct 31 01:05:30.223162 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223131 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223171 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223208 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223234 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d75beef-0864-46f3-9fab-1cc248c196a1-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223276 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223324 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223356 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223375 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223403 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwld\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-kube-api-access-mgwld\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223547 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.223630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223618 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.224108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223755 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.224108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.223821 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.224213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.224145 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d75beef-0864-46f3-9fab-1cc248c196a1-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.225556 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.225533 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.225817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.225800 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.283487 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.283428 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.283569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.283543 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwld\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-kube-api-access-mgwld\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.341125 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.341101 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:30.769463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.769424 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r"] Oct 31 01:05:30.775392 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:05:30.775361 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d75beef_0864_46f3_9fab_1cc248c196a1.slice/crio-3c009059304d026c75f2b3c1bc77e14d05cf4ce0059407dea9bfd899057e6459 WatchSource:0}: Error finding container 3c009059304d026c75f2b3c1bc77e14d05cf4ce0059407dea9bfd899057e6459: Status 404 returned error can't find the container with id 3c009059304d026c75f2b3c1bc77e14d05cf4ce0059407dea9bfd899057e6459 Oct 31 01:05:30.777745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.777704 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:05:30.777825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.777782 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:05:30.777825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:30.777814 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:05:31.510717 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:31.507121 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" event={"ID":"9d75beef-0864-46f3-9fab-1cc248c196a1","Type":"ContainerStarted","Data":"90e2199281974a65ea24d36023b8311f74fbaf7122e7735371c5c6ffbfe77e49"} Oct 31 01:05:31.510717 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:31.507223 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" event={"ID":"9d75beef-0864-46f3-9fab-1cc248c196a1","Type":"ContainerStarted","Data":"3c009059304d026c75f2b3c1bc77e14d05cf4ce0059407dea9bfd899057e6459"} Oct 31 01:05:31.684568 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:31.684511 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" podStartSLOduration=2.684493615 podStartE2EDuration="2.684493615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:31.6833129 +0000 UTC m=+2213.219455206" watchObservedRunningTime="2026-10-31 01:05:31.684493615 +0000 UTC m=+2213.220635920" Oct 31 01:05:32.341884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.341851 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:32.343133 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.343098 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:05:32.343280 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.343148 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 01:05:32.526011 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.525972 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6"] Oct 31 01:05:32.531127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.531103 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.582237 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.582213 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:32.584322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.584297 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6"] Oct 31 01:05:32.629815 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.629745 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:32.630028 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.629862 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:05:32.630428 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.630410 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:05:32.655426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.655397 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jswsl\" (UniqueName: \"kubernetes.io/projected/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-kube-api-access-jswsl\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.655647 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.655468 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.655647 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.655631 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.757110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.757079 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jswsl\" (UniqueName: \"kubernetes.io/projected/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-kube-api-access-jswsl\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.757268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.757140 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.757268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.757188 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.757819 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.757795 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.759648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.759626 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.827513 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.827479 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jswsl\" (UniqueName: \"kubernetes.io/projected/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-kube-api-access-jswsl\") pod \"maas-api-revoke-sub-14b41a8d-nw4p6\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:32.845346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:32.845317 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:33.033433 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:05:33.033398 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:05:33.241917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:33.241846 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6"] Oct 31 01:05:33.247404 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:05:33.247376 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a01bf40_450f_4ad4_bb34_3e5c16555ae4.slice/crio-0ba912af73245b4689f9b0495116032a436ce3777ca119de1ac332b7a2d44a06 WatchSource:0}: Error finding container 0ba912af73245b4689f9b0495116032a436ce3777ca119de1ac332b7a2d44a06: Status 404 returned error can't find the container with id 0ba912af73245b4689f9b0495116032a436ce3777ca119de1ac332b7a2d44a06 Oct 31 01:05:33.342407 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:33.342379 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:05:33.342579 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:33.342433 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 01:05:33.536732 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:33.536628 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" event={"ID":"5a01bf40-450f-4ad4-bb34-3e5c16555ae4","Type":"ContainerStarted","Data":"0ba912af73245b4689f9b0495116032a436ce3777ca119de1ac332b7a2d44a06"} Oct 31 01:05:34.341771 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:34.341745 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:05:34.341912 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:34.341792 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 01:05:34.545903 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:34.545873 5719 generic.go:356] "Generic (PLEG): container finished" podID="5a01bf40-450f-4ad4-bb34-3e5c16555ae4" containerID="7ee9476cb710b0cd73100b6b724eaafa6cdd5b267d8639f81f4151bc3ce4581f" exitCode=0 Oct 31 01:05:34.546292 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:34.545925 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" event={"ID":"5a01bf40-450f-4ad4-bb34-3e5c16555ae4","Type":"ContainerDied","Data":"7ee9476cb710b0cd73100b6b724eaafa6cdd5b267d8639f81f4151bc3ce4581f"} Oct 31 01:05:35.342333 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.342300 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:05:35.342491 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.342352 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 01:05:35.736859 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.736832 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:35.892520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.892454 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-openshift-ca\") pod \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " Oct 31 01:05:35.892520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.892496 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-xks-ca\") pod \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " Oct 31 01:05:35.892719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.892560 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-kube-api-access-jswsl\" (UniqueName: \"kubernetes.io/projected/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-kube-api-access-jswsl\") pod \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\" (UID: \"5a01bf40-450f-4ad4-bb34-3e5c16555ae4\") " Oct 31 01:05:35.892874 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.892850 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-openshift-ca" pod "5a01bf40-450f-4ad4-bb34-3e5c16555ae4" (UID: "5a01bf40-450f-4ad4-bb34-3e5c16555ae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:35.893093 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.893072 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:35.894706 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.894663 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-xks-ca" pod "5a01bf40-450f-4ad4-bb34-3e5c16555ae4" (UID: "5a01bf40-450f-4ad4-bb34-3e5c16555ae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:35.894803 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.894742 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-kube-api-access-jswsl" pod "5a01bf40-450f-4ad4-bb34-3e5c16555ae4" (UID: "5a01bf40-450f-4ad4-bb34-3e5c16555ae4"). InnerVolumeSpecName "kube-api-access-jswsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:35.994725 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.994677 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jswsl\" (UniqueName: \"kubernetes.io/projected/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-kube-api-access-jswsl\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:35.994725 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:35.994725 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a01bf40-450f-4ad4-bb34-3e5c16555ae4-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:36.341429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:36.341395 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:05:36.341616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:36.341444 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 01:05:36.561147 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:36.561118 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" Oct 31 01:05:36.561338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:36.561164 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6" event={"ID":"5a01bf40-450f-4ad4-bb34-3e5c16555ae4","Type":"ContainerDied","Data":"0ba912af73245b4689f9b0495116032a436ce3777ca119de1ac332b7a2d44a06"} Oct 31 01:05:36.561338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:36.561234 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ba912af73245b4689f9b0495116032a436ce3777ca119de1ac332b7a2d44a06" Oct 31 01:05:37.342425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:37.342383 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 01:05:37.342933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:37.342439 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 01:05:38.346174 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:38.346144 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:38.346577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:38.346527 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:38.347240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:38.347224 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:05:41.019395 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:41.019368 5719 scope.go:122] "RemoveContainer" containerID="89582cd562374a6b0713d7cedf7a0423d4826b377b7a838529238fd458e63688" Oct 31 01:05:49.044135 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.044100 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd"] Oct 31 01:05:49.044619 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.044603 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5a01bf40-450f-4ad4-bb34-3e5c16555ae4" containerName="revoke-keys" Oct 31 01:05:49.044681 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.044622 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a01bf40-450f-4ad4-bb34-3e5c16555ae4" containerName="revoke-keys" Oct 31 01:05:49.044759 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.044728 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a01bf40-450f-4ad4-bb34-3e5c16555ae4" containerName="revoke-keys" Oct 31 01:05:49.049579 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.049562 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.103749 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.103722 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd"] Oct 31 01:05:49.139862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.139834 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.139997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.139984 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669zm\" (UniqueName: \"kubernetes.io/projected/1cc14856-e90e-47c6-869e-4f7b07885e6b-kube-api-access-669zm\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.140039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.140027 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.148299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.148282 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:05:49.148352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.148343 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:49.148387 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.148345 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:49.148447 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.148433 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:05:49.240457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.240430 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669zm\" (UniqueName: \"kubernetes.io/projected/1cc14856-e90e-47c6-869e-4f7b07885e6b-kube-api-access-669zm\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.240574 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.240477 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.240574 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.240507 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.241094 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.241062 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.242989 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.242968 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.299183 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.299122 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-669zm\" (UniqueName: \"kubernetes.io/projected/1cc14856-e90e-47c6-869e-4f7b07885e6b-kube-api-access-669zm\") pod \"maas-api-revoke-sub-c325f7f4-ndcjd\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.362187 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.362166 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:49.733493 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:49.733464 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd"] Oct 31 01:05:49.738495 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:05:49.738469 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc14856_e90e_47c6_869e_4f7b07885e6b.slice/crio-538f6b0b402caca17ed38f8735b9a108d814d53e7a25814d6c922a598cccec24 WatchSource:0}: Error finding container 538f6b0b402caca17ed38f8735b9a108d814d53e7a25814d6c922a598cccec24: Status 404 returned error can't find the container with id 538f6b0b402caca17ed38f8735b9a108d814d53e7a25814d6c922a598cccec24 Oct 31 01:05:50.651092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:50.651063 5719 generic.go:356] "Generic (PLEG): container finished" podID="1cc14856-e90e-47c6-869e-4f7b07885e6b" containerID="f859837c36f5a874ed5431445869afdb2c302cdd9463a5084faa9e119fa3adca" exitCode=0 Oct 31 01:05:50.651455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:50.651155 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" event={"ID":"1cc14856-e90e-47c6-869e-4f7b07885e6b","Type":"ContainerDied","Data":"f859837c36f5a874ed5431445869afdb2c302cdd9463a5084faa9e119fa3adca"} Oct 31 01:05:50.651455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:50.651205 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" event={"ID":"1cc14856-e90e-47c6-869e-4f7b07885e6b","Type":"ContainerStarted","Data":"538f6b0b402caca17ed38f8735b9a108d814d53e7a25814d6c922a598cccec24"} Oct 31 01:05:51.799921 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.799899 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:51.866598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.866576 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-openshift-ca\") pod \"1cc14856-e90e-47c6-869e-4f7b07885e6b\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " Oct 31 01:05:51.866748 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.866644 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-xks-ca\") pod \"1cc14856-e90e-47c6-869e-4f7b07885e6b\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " Oct 31 01:05:51.866748 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.866741 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc14856-e90e-47c6-869e-4f7b07885e6b-kube-api-access-669zm\" (UniqueName: \"kubernetes.io/projected/1cc14856-e90e-47c6-869e-4f7b07885e6b-kube-api-access-669zm\") pod \"1cc14856-e90e-47c6-869e-4f7b07885e6b\" (UID: \"1cc14856-e90e-47c6-869e-4f7b07885e6b\") " Oct 31 01:05:51.867008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.866983 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-openshift-ca" pod "1cc14856-e90e-47c6-869e-4f7b07885e6b" (UID: "1cc14856-e90e-47c6-869e-4f7b07885e6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:05:51.867090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.867077 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:51.868803 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.868779 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-xks-ca" pod "1cc14856-e90e-47c6-869e-4f7b07885e6b" (UID: "1cc14856-e90e-47c6-869e-4f7b07885e6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:05:51.868916 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.868882 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc14856-e90e-47c6-869e-4f7b07885e6b-kube-api-access-669zm" pod "1cc14856-e90e-47c6-869e-4f7b07885e6b" (UID: "1cc14856-e90e-47c6-869e-4f7b07885e6b"). InnerVolumeSpecName "kube-api-access-669zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:05:51.968409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.968338 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc14856-e90e-47c6-869e-4f7b07885e6b-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:51.968409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:51.968367 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-669zm\" (UniqueName: \"kubernetes.io/projected/1cc14856-e90e-47c6-869e-4f7b07885e6b-kube-api-access-669zm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:05:52.663374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:52.663345 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" Oct 31 01:05:52.663546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:52.663381 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd" event={"ID":"1cc14856-e90e-47c6-869e-4f7b07885e6b","Type":"ContainerDied","Data":"538f6b0b402caca17ed38f8735b9a108d814d53e7a25814d6c922a598cccec24"} Oct 31 01:05:52.663546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:52.663436 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538f6b0b402caca17ed38f8735b9a108d814d53e7a25814d6c922a598cccec24" Oct 31 01:05:56.303404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:56.303365 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:05:59.847267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.847232 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk"] Oct 31 01:05:59.847687 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.847674 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1cc14856-e90e-47c6-869e-4f7b07885e6b" containerName="revoke-keys" Oct 31 01:05:59.847761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.847701 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc14856-e90e-47c6-869e-4f7b07885e6b" containerName="revoke-keys" Oct 31 01:05:59.847810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.847799 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc14856-e90e-47c6-869e-4f7b07885e6b" containerName="revoke-keys" Oct 31 01:05:59.851018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.850982 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:05:59.908212 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.908179 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk"] Oct 31 01:05:59.951887 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.951682 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:05:59.951887 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.951812 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:05:59.952078 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.951897 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5n2\" (UniqueName: \"kubernetes.io/projected/5fa25b9f-0c3a-4684-a48c-46440d264d57-kube-api-access-gw5n2\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:05:59.952438 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.952364 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:05:59.952438 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.952369 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:05:59.952438 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.952409 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:05:59.952438 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:05:59.952433 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:00.053008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.052974 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:00.053206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.053020 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:00.053206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.053066 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5n2\" (UniqueName: \"kubernetes.io/projected/5fa25b9f-0c3a-4684-a48c-46440d264d57-kube-api-access-gw5n2\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:00.053912 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.053808 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:00.055545 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.055523 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:00.112517 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.112451 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5n2\" (UniqueName: \"kubernetes.io/projected/5fa25b9f-0c3a-4684-a48c-46440d264d57-kube-api-access-gw5n2\") pod \"maas-api-revoke-sub-86117516-6ndpk\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:00.164348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.164320 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:00.540417 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.540349 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk"] Oct 31 01:06:00.546344 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:00.546321 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa25b9f_0c3a_4684_a48c_46440d264d57.slice/crio-a254a45e1a39806e27c01734a16c815b7895f57a787e51a808e0d0f60e52eb6f WatchSource:0}: Error finding container a254a45e1a39806e27c01734a16c815b7895f57a787e51a808e0d0f60e52eb6f: Status 404 returned error can't find the container with id a254a45e1a39806e27c01734a16c815b7895f57a787e51a808e0d0f60e52eb6f Oct 31 01:06:00.712547 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:00.712516 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" event={"ID":"5fa25b9f-0c3a-4684-a48c-46440d264d57","Type":"ContainerStarted","Data":"a254a45e1a39806e27c01734a16c815b7895f57a787e51a808e0d0f60e52eb6f"} Oct 31 01:06:01.513539 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.513510 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46"] Oct 31 01:06:01.517127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.517102 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.569940 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.569917 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knm22\" (UniqueName: \"kubernetes.io/projected/8a33a497-428c-4de7-921f-c3f48c1b1e4f-kube-api-access-knm22\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.570090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.569976 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.570166 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.570128 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.574118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.574095 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46"] Oct 31 01:06:01.671127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.671061 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.671240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.671158 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.671240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.671189 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knm22\" (UniqueName: \"kubernetes.io/projected/8a33a497-428c-4de7-921f-c3f48c1b1e4f-kube-api-access-knm22\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.671809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.671790 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.673631 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.673616 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.719796 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.719766 5719 generic.go:356] "Generic (PLEG): container finished" podID="5fa25b9f-0c3a-4684-a48c-46440d264d57" containerID="57e3f6f1a93373167645b2ac47dcbfaeab82445b5dd9fefd26200c4302e79e6e" exitCode=0 Oct 31 01:06:01.719904 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.719829 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" event={"ID":"5fa25b9f-0c3a-4684-a48c-46440d264d57","Type":"ContainerDied","Data":"57e3f6f1a93373167645b2ac47dcbfaeab82445b5dd9fefd26200c4302e79e6e"} Oct 31 01:06:01.730240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.730209 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knm22\" (UniqueName: \"kubernetes.io/projected/8a33a497-428c-4de7-921f-c3f48c1b1e4f-kube-api-access-knm22\") pod \"maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:01.829505 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:01.829484 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:02.202647 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.202618 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46"] Oct 31 01:06:02.207151 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:02.207126 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a33a497_428c_4de7_921f_c3f48c1b1e4f.slice/crio-bbb0d77edf4ad472f404e7c5fbc3f66461fc7bdddb3983c5f868ce693c7419c2 WatchSource:0}: Error finding container bbb0d77edf4ad472f404e7c5fbc3f66461fc7bdddb3983c5f868ce693c7419c2: Status 404 returned error can't find the container with id bbb0d77edf4ad472f404e7c5fbc3f66461fc7bdddb3983c5f868ce693c7419c2 Oct 31 01:06:02.727457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.727429 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" event={"ID":"8a33a497-428c-4de7-921f-c3f48c1b1e4f","Type":"ContainerStarted","Data":"ea6cd56caf034ba42e35003da822e7cb88dda4ef184faecb480ae0112956c8d3"} Oct 31 01:06:02.727893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.727474 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" event={"ID":"8a33a497-428c-4de7-921f-c3f48c1b1e4f","Type":"ContainerStarted","Data":"bbb0d77edf4ad472f404e7c5fbc3f66461fc7bdddb3983c5f868ce693c7419c2"} Oct 31 01:06:02.875252 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.875230 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:02.898920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.898881 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" podStartSLOduration=1.898869117 podStartE2EDuration="1.898869117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:02.850478474 +0000 UTC m=+2244.386620781" watchObservedRunningTime="2026-10-31 01:06:02.898869117 +0000 UTC m=+2244.435011446" Oct 31 01:06:02.983050 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.983020 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-xks-ca\") pod \"5fa25b9f-0c3a-4684-a48c-46440d264d57\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " Oct 31 01:06:02.983255 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.983165 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa25b9f-0c3a-4684-a48c-46440d264d57-kube-api-access-gw5n2\" (UniqueName: \"kubernetes.io/projected/5fa25b9f-0c3a-4684-a48c-46440d264d57-kube-api-access-gw5n2\") pod \"5fa25b9f-0c3a-4684-a48c-46440d264d57\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " Oct 31 01:06:02.983335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.983263 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-openshift-ca\") pod \"5fa25b9f-0c3a-4684-a48c-46440d264d57\" (UID: \"5fa25b9f-0c3a-4684-a48c-46440d264d57\") " Oct 31 01:06:02.983729 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.983684 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-openshift-ca" pod "5fa25b9f-0c3a-4684-a48c-46440d264d57" (UID: "5fa25b9f-0c3a-4684-a48c-46440d264d57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:02.985943 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.985919 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-xks-ca" pod "5fa25b9f-0c3a-4684-a48c-46440d264d57" (UID: "5fa25b9f-0c3a-4684-a48c-46440d264d57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:02.986129 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:02.986103 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa25b9f-0c3a-4684-a48c-46440d264d57-kube-api-access-gw5n2" pod "5fa25b9f-0c3a-4684-a48c-46440d264d57" (UID: "5fa25b9f-0c3a-4684-a48c-46440d264d57"). InnerVolumeSpecName "kube-api-access-gw5n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:03.013418 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:06:03.013381 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:06:03.085207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:03.085180 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5n2\" (UniqueName: \"kubernetes.io/projected/5fa25b9f-0c3a-4684-a48c-46440d264d57-kube-api-access-gw5n2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:03.085327 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:03.085210 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:03.085327 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:03.085228 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa25b9f-0c3a-4684-a48c-46440d264d57-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:03.734991 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:03.734956 5719 generic.go:356] "Generic (PLEG): container finished" podID="8a33a497-428c-4de7-921f-c3f48c1b1e4f" containerID="ea6cd56caf034ba42e35003da822e7cb88dda4ef184faecb480ae0112956c8d3" exitCode=0 Oct 31 01:06:03.735418 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:03.735043 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" event={"ID":"8a33a497-428c-4de7-921f-c3f48c1b1e4f","Type":"ContainerDied","Data":"ea6cd56caf034ba42e35003da822e7cb88dda4ef184faecb480ae0112956c8d3"} Oct 31 01:06:03.736396 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:03.736375 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" event={"ID":"5fa25b9f-0c3a-4684-a48c-46440d264d57","Type":"ContainerDied","Data":"a254a45e1a39806e27c01734a16c815b7895f57a787e51a808e0d0f60e52eb6f"} Oct 31 01:06:03.736396 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:03.736388 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk" Oct 31 01:06:03.736579 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:03.736410 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a254a45e1a39806e27c01734a16c815b7895f57a787e51a808e0d0f60e52eb6f" Oct 31 01:06:04.887063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:04.887037 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:05.004016 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.003950 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-xks-ca\") pod \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " Oct 31 01:06:05.004162 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.004094 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a33a497-428c-4de7-921f-c3f48c1b1e4f-kube-api-access-knm22\" (UniqueName: \"kubernetes.io/projected/8a33a497-428c-4de7-921f-c3f48c1b1e4f-kube-api-access-knm22\") pod \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " Oct 31 01:06:05.004227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.004168 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-openshift-ca\") pod \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\" (UID: \"8a33a497-428c-4de7-921f-c3f48c1b1e4f\") " Oct 31 01:06:05.004501 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.004474 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-openshift-ca" pod "8a33a497-428c-4de7-921f-c3f48c1b1e4f" (UID: "8a33a497-428c-4de7-921f-c3f48c1b1e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:05.004643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.004608 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:05.006050 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.006023 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-xks-ca" pod "8a33a497-428c-4de7-921f-c3f48c1b1e4f" (UID: "8a33a497-428c-4de7-921f-c3f48c1b1e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:05.006400 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.006379 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a33a497-428c-4de7-921f-c3f48c1b1e4f-kube-api-access-knm22" pod "8a33a497-428c-4de7-921f-c3f48c1b1e4f" (UID: "8a33a497-428c-4de7-921f-c3f48c1b1e4f"). InnerVolumeSpecName "kube-api-access-knm22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:05.105685 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.105663 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a33a497-428c-4de7-921f-c3f48c1b1e4f-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:05.105821 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.105685 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knm22\" (UniqueName: \"kubernetes.io/projected/8a33a497-428c-4de7-921f-c3f48c1b1e4f-kube-api-access-knm22\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:05.749429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.749398 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" event={"ID":"8a33a497-428c-4de7-921f-c3f48c1b1e4f","Type":"ContainerDied","Data":"bbb0d77edf4ad472f404e7c5fbc3f66461fc7bdddb3983c5f868ce693c7419c2"} Oct 31 01:06:05.749429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.749430 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46" Oct 31 01:06:05.749645 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:05.749444 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbb0d77edf4ad472f404e7c5fbc3f66461fc7bdddb3983c5f868ce693c7419c2" Oct 31 01:06:16.295165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.295075 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f"] Oct 31 01:06:16.295796 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.295775 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8a33a497-428c-4de7-921f-c3f48c1b1e4f" containerName="revoke-keys" Oct 31 01:06:16.295891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.295797 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a33a497-428c-4de7-921f-c3f48c1b1e4f" containerName="revoke-keys" Oct 31 01:06:16.295891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.295816 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5fa25b9f-0c3a-4684-a48c-46440d264d57" containerName="revoke-keys" Oct 31 01:06:16.295891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.295825 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa25b9f-0c3a-4684-a48c-46440d264d57" containerName="revoke-keys" Oct 31 01:06:16.296068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.296000 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a33a497-428c-4de7-921f-c3f48c1b1e4f" containerName="revoke-keys" Oct 31 01:06:16.296068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.296017 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa25b9f-0c3a-4684-a48c-46440d264d57" containerName="revoke-keys" Oct 31 01:06:16.299348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.299328 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.353039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.353019 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:16.353166 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.353042 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:16.355842 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.355817 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f"] Oct 31 01:06:16.399842 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.399819 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:16.399842 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.399823 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:06:16.419537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.419515 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwb5\" (UniqueName: \"kubernetes.io/projected/d1210ae8-451c-4438-9e07-135e30dbb333-kube-api-access-xcwb5\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.419633 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.419565 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.419678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.419645 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.521196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.521169 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.521306 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.521284 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwb5\" (UniqueName: \"kubernetes.io/projected/d1210ae8-451c-4438-9e07-135e30dbb333-kube-api-access-xcwb5\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.521358 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.521348 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.521836 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.521818 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.523740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.523715 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-xks-ca\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.581525 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.581495 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwb5\" (UniqueName: \"kubernetes.io/projected/d1210ae8-451c-4438-9e07-135e30dbb333-kube-api-access-xcwb5\") pod \"maas-api-revoke-sub-682cb3fc-gfs4f\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:16.612545 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:16.612518 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:17.024471 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:17.024396 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f"] Oct 31 01:06:17.029831 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:17.029804 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1210ae8_451c_4438_9e07_135e30dbb333.slice/crio-d2b28a28076be36797705217c9cb0598277dd7bd06207aca017ec2fb55ba3a20 WatchSource:0}: Error finding container d2b28a28076be36797705217c9cb0598277dd7bd06207aca017ec2fb55ba3a20: Status 404 returned error can't find the container with id d2b28a28076be36797705217c9cb0598277dd7bd06207aca017ec2fb55ba3a20 Oct 31 01:06:17.820967 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:17.820933 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" event={"ID":"d1210ae8-451c-4438-9e07-135e30dbb333","Type":"ContainerStarted","Data":"85318a549f2871ee1267298b6c7dcedaebf5fcc849835525215e92b637a285cd"} Oct 31 01:06:17.821346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:17.820987 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" event={"ID":"d1210ae8-451c-4438-9e07-135e30dbb333","Type":"ContainerStarted","Data":"d2b28a28076be36797705217c9cb0598277dd7bd06207aca017ec2fb55ba3a20"} Oct 31 01:06:17.934955 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:17.934906 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" podStartSLOduration=1.934894451 podStartE2EDuration="1.934894451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:17.934811355 +0000 UTC m=+2259.470953661" watchObservedRunningTime="2026-10-31 01:06:17.934894451 +0000 UTC m=+2259.471036770" Oct 31 01:06:18.830087 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:18.830041 5719 generic.go:356] "Generic (PLEG): container finished" podID="d1210ae8-451c-4438-9e07-135e30dbb333" containerID="85318a549f2871ee1267298b6c7dcedaebf5fcc849835525215e92b637a285cd" exitCode=0 Oct 31 01:06:18.830511 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:18.830209 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" event={"ID":"d1210ae8-451c-4438-9e07-135e30dbb333","Type":"ContainerDied","Data":"85318a549f2871ee1267298b6c7dcedaebf5fcc849835525215e92b637a285cd"} Oct 31 01:06:19.310094 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.310014 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl"] Oct 31 01:06:19.313799 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.313779 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:19.367266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.367246 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3f25cd\"" Oct 31 01:06:19.370761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.368398 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3f25cd-dockercfg-9n9w9\"" Oct 31 01:06:19.370761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.370643 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl"] Oct 31 01:06:19.454283 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.454254 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1d3962-918c-480b-8ab0-35d114504e31-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl\" (UID: \"1d1d3962-918c-480b-8ab0-35d114504e31\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:19.454485 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.454322 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdhh\" (UniqueName: \"kubernetes.io/projected/1d1d3962-918c-480b-8ab0-35d114504e31-kube-api-access-gjdhh\") pod \"payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl\" (UID: \"1d1d3962-918c-480b-8ab0-35d114504e31\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:19.474225 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.474197 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh"] Oct 31 01:06:19.478051 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.478035 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:19.533798 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.533758 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh"] Oct 31 01:06:19.555914 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.555875 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1d3962-918c-480b-8ab0-35d114504e31-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl\" (UID: \"1d1d3962-918c-480b-8ab0-35d114504e31\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:19.556043 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.555938 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdhh\" (UniqueName: \"kubernetes.io/projected/1d1d3962-918c-480b-8ab0-35d114504e31-kube-api-access-gjdhh\") pod \"payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl\" (UID: \"1d1d3962-918c-480b-8ab0-35d114504e31\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:19.556535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.556514 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1d3962-918c-480b-8ab0-35d114504e31-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl\" (UID: \"1d1d3962-918c-480b-8ab0-35d114504e31\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:19.615468 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.615436 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdhh\" (UniqueName: \"kubernetes.io/projected/1d1d3962-918c-480b-8ab0-35d114504e31-kube-api-access-gjdhh\") pod \"payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl\" (UID: \"1d1d3962-918c-480b-8ab0-35d114504e31\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:19.626309 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.626287 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:19.656656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.656628 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvfj\" (UniqueName: \"kubernetes.io/projected/f81e8be5-748b-4a9f-98c3-909da65cf65a-kube-api-access-9rvfj\") pod \"payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh\" (UID: \"f81e8be5-748b-4a9f-98c3-909da65cf65a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:19.656815 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.656672 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81e8be5-748b-4a9f-98c3-909da65cf65a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh\" (UID: \"f81e8be5-748b-4a9f-98c3-909da65cf65a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:19.758546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.758505 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvfj\" (UniqueName: \"kubernetes.io/projected/f81e8be5-748b-4a9f-98c3-909da65cf65a-kube-api-access-9rvfj\") pod \"payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh\" (UID: \"f81e8be5-748b-4a9f-98c3-909da65cf65a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:19.758770 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.758555 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81e8be5-748b-4a9f-98c3-909da65cf65a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh\" (UID: \"f81e8be5-748b-4a9f-98c3-909da65cf65a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:19.759267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.759249 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81e8be5-748b-4a9f-98c3-909da65cf65a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh\" (UID: \"f81e8be5-748b-4a9f-98c3-909da65cf65a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:19.817514 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.817487 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvfj\" (UniqueName: \"kubernetes.io/projected/f81e8be5-748b-4a9f-98c3-909da65cf65a-kube-api-access-9rvfj\") pod \"payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh\" (UID: \"f81e8be5-748b-4a9f-98c3-909da65cf65a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:19.973933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:19.973905 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:20.001863 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.001832 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl"] Oct 31 01:06:20.009581 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:20.009553 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1d3962_918c_480b_8ab0_35d114504e31.slice/crio-8a1ad46efa74df76b86da35f29916a536e41e4c19f7fcb4bfd3987fe7f793ac7 WatchSource:0}: Error finding container 8a1ad46efa74df76b86da35f29916a536e41e4c19f7fcb4bfd3987fe7f793ac7: Status 404 returned error can't find the container with id 8a1ad46efa74df76b86da35f29916a536e41e4c19f7fcb4bfd3987fe7f793ac7 Oct 31 01:06:20.062276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.062257 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1210ae8-451c-4438-9e07-135e30dbb333-kube-api-access-xcwb5\" (UniqueName: \"kubernetes.io/projected/d1210ae8-451c-4438-9e07-135e30dbb333-kube-api-access-xcwb5\") pod \"d1210ae8-451c-4438-9e07-135e30dbb333\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " Oct 31 01:06:20.062361 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.062345 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-openshift-ca\") pod \"d1210ae8-451c-4438-9e07-135e30dbb333\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " Oct 31 01:06:20.062591 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.062406 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-xks-ca\") pod \"d1210ae8-451c-4438-9e07-135e30dbb333\" (UID: \"d1210ae8-451c-4438-9e07-135e30dbb333\") " Oct 31 01:06:20.062757 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.062731 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-openshift-ca" pod "d1210ae8-451c-4438-9e07-135e30dbb333" (UID: "d1210ae8-451c-4438-9e07-135e30dbb333"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:20.062932 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.062905 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:20.064356 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.064332 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1210ae8-451c-4438-9e07-135e30dbb333-kube-api-access-xcwb5" pod "d1210ae8-451c-4438-9e07-135e30dbb333" (UID: "d1210ae8-451c-4438-9e07-135e30dbb333"). InnerVolumeSpecName "kube-api-access-xcwb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:20.064487 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.064466 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-xks-ca" pod "d1210ae8-451c-4438-9e07-135e30dbb333" (UID: "d1210ae8-451c-4438-9e07-135e30dbb333"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:20.091262 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.091239 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:20.164737 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.164704 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1210ae8-451c-4438-9e07-135e30dbb333-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:20.164848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.164743 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcwb5\" (UniqueName: \"kubernetes.io/projected/d1210ae8-451c-4438-9e07-135e30dbb333-kube-api-access-xcwb5\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:20.466763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.466636 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh"] Oct 31 01:06:20.471150 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:20.471124 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81e8be5_748b_4a9f_98c3_909da65cf65a.slice/crio-d3fd29138c7c5576b32661ae870626dbabe15e902bce52ba74b70b628a26c35f WatchSource:0}: Error finding container d3fd29138c7c5576b32661ae870626dbabe15e902bce52ba74b70b628a26c35f: Status 404 returned error can't find the container with id d3fd29138c7c5576b32661ae870626dbabe15e902bce52ba74b70b628a26c35f Oct 31 01:06:20.842514 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.842485 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" Oct 31 01:06:20.842738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.842534 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f" event={"ID":"d1210ae8-451c-4438-9e07-135e30dbb333","Type":"ContainerDied","Data":"d2b28a28076be36797705217c9cb0598277dd7bd06207aca017ec2fb55ba3a20"} Oct 31 01:06:20.842738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.842592 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2b28a28076be36797705217c9cb0598277dd7bd06207aca017ec2fb55ba3a20" Oct 31 01:06:20.844325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.844298 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" event={"ID":"1d1d3962-918c-480b-8ab0-35d114504e31","Type":"ContainerStarted","Data":"2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639"} Oct 31 01:06:20.844444 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.844345 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" event={"ID":"1d1d3962-918c-480b-8ab0-35d114504e31","Type":"ContainerStarted","Data":"8a1ad46efa74df76b86da35f29916a536e41e4c19f7fcb4bfd3987fe7f793ac7"} Oct 31 01:06:20.844444 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.844425 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:20.846182 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.846161 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" event={"ID":"f81e8be5-748b-4a9f-98c3-909da65cf65a","Type":"ContainerStarted","Data":"e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09"} Oct 31 01:06:20.846293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.846194 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" event={"ID":"f81e8be5-748b-4a9f-98c3-909da65cf65a","Type":"ContainerStarted","Data":"d3fd29138c7c5576b32661ae870626dbabe15e902bce52ba74b70b628a26c35f"} Oct 31 01:06:20.846293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.846223 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:20.959555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:20.959506 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" podStartSLOduration=1.9594908549999999 podStartE2EDuration="1.959490855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:20.956393967 +0000 UTC m=+2262.492536272" watchObservedRunningTime="2026-10-31 01:06:20.959490855 +0000 UTC m=+2262.495633178" Oct 31 01:06:21.072070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:21.071996 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" podStartSLOduration=2.071973991 podStartE2EDuration="2.071973991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:21.06831287 +0000 UTC m=+2262.604455177" watchObservedRunningTime="2026-10-31 01:06:21.071973991 +0000 UTC m=+2262.608116298" Oct 31 01:06:29.538073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.538044 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc"] Oct 31 01:06:29.538577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.538561 5719 cpu_manager.go:414] "containerMap: removing container" podUID="d1210ae8-451c-4438-9e07-135e30dbb333" containerName="revoke-keys" Oct 31 01:06:29.538659 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.538580 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1210ae8-451c-4438-9e07-135e30dbb333" containerName="revoke-keys" Oct 31 01:06:29.538725 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.538706 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1210ae8-451c-4438-9e07-135e30dbb333" containerName="revoke-keys" Oct 31 01:06:29.541869 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.541849 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.558137 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.558111 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.558251 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.558167 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcns9\" (UniqueName: \"kubernetes.io/projected/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-kube-api-access-kcns9\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.558324 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.558299 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.643498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.643468 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:29.643498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.643497 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:29.643741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.643503 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:29.643741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.643650 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:06:29.646183 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.646164 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc"] Oct 31 01:06:29.659068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.659038 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.659243 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.659148 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcns9\" (UniqueName: \"kubernetes.io/projected/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-kube-api-access-kcns9\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.659329 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.659267 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.659793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.659773 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.661990 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.661957 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.718457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.718428 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcns9\" (UniqueName: \"kubernetes.io/projected/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-kube-api-access-kcns9\") pod \"maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:29.855901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:29.855866 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:30.237044 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:30.236963 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc"] Oct 31 01:06:30.241578 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:30.241552 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66c4055_a8e5_43ee_ad5a_41a14a7222ee.slice/crio-0366a29e401c5cd1600ffcc896a28e65ddaa3b9e0af6f6d0eeb2d2e3507651bf WatchSource:0}: Error finding container 0366a29e401c5cd1600ffcc896a28e65ddaa3b9e0af6f6d0eeb2d2e3507651bf: Status 404 returned error can't find the container with id 0366a29e401c5cd1600ffcc896a28e65ddaa3b9e0af6f6d0eeb2d2e3507651bf Oct 31 01:06:30.604892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:30.604863 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46"] Oct 31 01:06:30.607523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:30.607500 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-7ea4b0-7pd46"] Oct 31 01:06:30.912628 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:30.912597 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" event={"ID":"a66c4055-a8e5-43ee-ad5a-41a14a7222ee","Type":"ContainerStarted","Data":"5c61033f6ab46c30e3856da48b9fcdd5525407e675c7a3001a79a512983b3714"} Oct 31 01:06:30.912858 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:30.912646 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" event={"ID":"a66c4055-a8e5-43ee-ad5a-41a14a7222ee","Type":"ContainerStarted","Data":"0366a29e401c5cd1600ffcc896a28e65ddaa3b9e0af6f6d0eeb2d2e3507651bf"} Oct 31 01:06:31.029493 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:31.029387 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" podStartSLOduration=2.029360913 podStartE2EDuration="2.029360913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:31.023973591 +0000 UTC m=+2272.560115907" watchObservedRunningTime="2026-10-31 01:06:31.029360913 +0000 UTC m=+2272.565503219" Oct 31 01:06:31.065940 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:31.065905 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh"] Oct 31 01:06:31.074158 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:31.074134 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-106dbcf1-bgkrh"] Oct 31 01:06:31.856525 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:31.856490 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:06:31.857043 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:31.856567 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:06:31.919996 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:31.919965 5719 generic.go:356] "Generic (PLEG): container finished" podID="a66c4055-a8e5-43ee-ad5a-41a14a7222ee" containerID="5c61033f6ab46c30e3856da48b9fcdd5525407e675c7a3001a79a512983b3714" exitCode=0 Oct 31 01:06:31.920176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:31.920044 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" event={"ID":"a66c4055-a8e5-43ee-ad5a-41a14a7222ee","Type":"ContainerDied","Data":"5c61033f6ab46c30e3856da48b9fcdd5525407e675c7a3001a79a512983b3714"} Oct 31 01:06:32.605449 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:32.605417 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26076112-5134-4aad-a830-6c047abca282" path="/var/lib/kubelet/pods/26076112-5134-4aad-a830-6c047abca282/volumes" Oct 31 01:06:32.605898 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:32.605879 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a33a497-428c-4de7-921f-c3f48c1b1e4f" path="/var/lib/kubelet/pods/8a33a497-428c-4de7-921f-c3f48c1b1e4f/volumes" Oct 31 01:06:33.019089 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:06:33.019027 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:06:33.075092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.075060 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:33.114373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.114347 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-openshift-ca\") pod \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " Oct 31 01:06:33.114515 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.114456 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-xks-ca\") pod \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " Oct 31 01:06:33.114601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.114582 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-kube-api-access-kcns9\" (UniqueName: \"kubernetes.io/projected/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-kube-api-access-kcns9\") pod \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\" (UID: \"a66c4055-a8e5-43ee-ad5a-41a14a7222ee\") " Oct 31 01:06:33.114676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.114650 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-openshift-ca" pod "a66c4055-a8e5-43ee-ad5a-41a14a7222ee" (UID: "a66c4055-a8e5-43ee-ad5a-41a14a7222ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:33.115006 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.114989 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:33.116567 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.116544 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-xks-ca" pod "a66c4055-a8e5-43ee-ad5a-41a14a7222ee" (UID: "a66c4055-a8e5-43ee-ad5a-41a14a7222ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:33.116754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.116738 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-kube-api-access-kcns9" pod "a66c4055-a8e5-43ee-ad5a-41a14a7222ee" (UID: "a66c4055-a8e5-43ee-ad5a-41a14a7222ee"). InnerVolumeSpecName "kube-api-access-kcns9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:33.216426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.216339 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcns9\" (UniqueName: \"kubernetes.io/projected/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-kube-api-access-kcns9\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:33.216426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.216367 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c4055-a8e5-43ee-ad5a-41a14a7222ee-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:33.934674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.934636 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" event={"ID":"a66c4055-a8e5-43ee-ad5a-41a14a7222ee","Type":"ContainerDied","Data":"0366a29e401c5cd1600ffcc896a28e65ddaa3b9e0af6f6d0eeb2d2e3507651bf"} Oct 31 01:06:33.934880 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.934648 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc" Oct 31 01:06:33.934880 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:33.934707 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0366a29e401c5cd1600ffcc896a28e65ddaa3b9e0af6f6d0eeb2d2e3507651bf" Oct 31 01:06:38.405515 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.405483 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk"] Oct 31 01:06:38.406003 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.405974 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a66c4055-a8e5-43ee-ad5a-41a14a7222ee" containerName="revoke-keys" Oct 31 01:06:38.406003 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.405986 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a66c4055-a8e5-43ee-ad5a-41a14a7222ee" containerName="revoke-keys" Oct 31 01:06:38.406106 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.406092 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a66c4055-a8e5-43ee-ad5a-41a14a7222ee" containerName="revoke-keys" Oct 31 01:06:38.409648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.409622 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.471839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.471801 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8lm9\" (UniqueName: \"kubernetes.io/projected/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-kube-api-access-s8lm9\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.472021 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.471974 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.472100 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.472076 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.511360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.511334 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:38.511360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.511356 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:38.511577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.511363 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:38.511577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.511465 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:06:38.514043 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.514027 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk"] Oct 31 01:06:38.566516 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.566483 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr"] Oct 31 01:06:38.570391 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.570371 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.572937 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.572917 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.573018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.572974 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.573089 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.573074 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8lm9\" (UniqueName: \"kubernetes.io/projected/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-kube-api-access-s8lm9\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.622139 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.622108 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:38.622286 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.622227 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:38.623798 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.623773 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.626194 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.626168 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.628835 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.628812 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr"] Oct 31 01:06:38.674015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.673945 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.674015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.673977 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.674193 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.674173 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj9mx\" (UniqueName: \"kubernetes.io/projected/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-kube-api-access-jj9mx\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.680068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.680036 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:38.683899 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.683871 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8lm9\" (UniqueName: \"kubernetes.io/projected/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-kube-api-access-s8lm9\") pod \"maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.771972 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.771945 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:06:38.772920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.772893 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:38.775569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.775544 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj9mx\" (UniqueName: \"kubernetes.io/projected/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-kube-api-access-jj9mx\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.775678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.775627 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.775678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.775659 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.776371 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.776350 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.778135 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.778109 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.835981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.835952 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj9mx\" (UniqueName: \"kubernetes.io/projected/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-kube-api-access-jj9mx\") pod \"maas-api-revoke-sub-40626f1b-bg8pr\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:38.882934 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:38.882899 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:39.146103 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:39.146067 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk"] Oct 31 01:06:39.150761 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:39.150730 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdb1ac5_ca09_40dd_a8b7_a5cff4cbc300.slice/crio-f9172f2fa3e8adaaf1d62efc48ce8a0d1451764c0ac8a02f183b4d7d91520e3b WatchSource:0}: Error finding container f9172f2fa3e8adaaf1d62efc48ce8a0d1451764c0ac8a02f183b4d7d91520e3b: Status 404 returned error can't find the container with id f9172f2fa3e8adaaf1d62efc48ce8a0d1451764c0ac8a02f183b4d7d91520e3b Oct 31 01:06:39.261302 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:39.261213 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr"] Oct 31 01:06:39.266101 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:39.266071 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cd0b66_ddf4_43e0_8d72_963bd5ae1f5d.slice/crio-5b392d57f50771416026a668c7a69b067caddab6054082f1c2cae5cad3cddb01 WatchSource:0}: Error finding container 5b392d57f50771416026a668c7a69b067caddab6054082f1c2cae5cad3cddb01: Status 404 returned error can't find the container with id 5b392d57f50771416026a668c7a69b067caddab6054082f1c2cae5cad3cddb01 Oct 31 01:06:39.980225 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:39.980194 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" event={"ID":"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300","Type":"ContainerStarted","Data":"7417d083d95c5f03553716e707b985db9c42d7798d7d34a96e1972a2bd31e054"} Oct 31 01:06:39.980705 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:39.980245 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" event={"ID":"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300","Type":"ContainerStarted","Data":"f9172f2fa3e8adaaf1d62efc48ce8a0d1451764c0ac8a02f183b4d7d91520e3b"} Oct 31 01:06:39.981872 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:39.981850 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" event={"ID":"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d","Type":"ContainerStarted","Data":"247adeeb9281f7835ab0115f9ec70a82a8d518babc8cb9ea5f9df709b19dea73"} Oct 31 01:06:39.981957 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:39.981889 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" event={"ID":"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d","Type":"ContainerStarted","Data":"5b392d57f50771416026a668c7a69b067caddab6054082f1c2cae5cad3cddb01"} Oct 31 01:06:40.093816 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:40.093762 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" podStartSLOduration=2.093748032 podStartE2EDuration="2.093748032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:40.091383015 +0000 UTC m=+2281.627525304" watchObservedRunningTime="2026-10-31 01:06:40.093748032 +0000 UTC m=+2281.629890336" Oct 31 01:06:40.205660 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:40.205606 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" podStartSLOduration=2.205590602 podStartE2EDuration="2.205590602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:40.202677794 +0000 UTC m=+2281.738820099" watchObservedRunningTime="2026-10-31 01:06:40.205590602 +0000 UTC m=+2281.741732906" Oct 31 01:06:40.990117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:40.990079 5719 generic.go:356] "Generic (PLEG): container finished" podID="fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300" containerID="7417d083d95c5f03553716e707b985db9c42d7798d7d34a96e1972a2bd31e054" exitCode=0 Oct 31 01:06:40.990583 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:40.990172 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" event={"ID":"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300","Type":"ContainerDied","Data":"7417d083d95c5f03553716e707b985db9c42d7798d7d34a96e1972a2bd31e054"} Oct 31 01:06:40.992023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:40.992001 5719 generic.go:356] "Generic (PLEG): container finished" podID="69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d" containerID="247adeeb9281f7835ab0115f9ec70a82a8d518babc8cb9ea5f9df709b19dea73" exitCode=0 Oct 31 01:06:40.992168 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:40.992068 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" event={"ID":"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d","Type":"ContainerDied","Data":"247adeeb9281f7835ab0115f9ec70a82a8d518babc8cb9ea5f9df709b19dea73"} Oct 31 01:06:41.051962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:41.051926 5719 scope.go:122] "RemoveContainer" containerID="769a03c14486e624e1876647ca129b7f0e431e176bb2992426640382f91495c2" Oct 31 01:06:42.169455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.169431 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:42.171600 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.171583 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:42.315573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.315497 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-xks-ca\") pod \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " Oct 31 01:06:42.315573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.315561 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-kube-api-access-s8lm9\" (UniqueName: \"kubernetes.io/projected/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-kube-api-access-s8lm9\") pod \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " Oct 31 01:06:42.315800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.315669 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-openshift-ca\") pod \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " Oct 31 01:06:42.315800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.315751 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-openshift-ca\") pod \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " Oct 31 01:06:42.315900 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.315809 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-kube-api-access-jj9mx\" (UniqueName: \"kubernetes.io/projected/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-kube-api-access-jj9mx\") pod \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\" (UID: \"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d\") " Oct 31 01:06:42.315972 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.315938 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-xks-ca\") pod \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\" (UID: \"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300\") " Oct 31 01:06:42.316071 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.316038 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-openshift-ca" pod "69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d" (UID: "69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:42.316148 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.316039 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-openshift-ca" pod "fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300" (UID: "fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:42.316587 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.316564 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:42.316722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.316593 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:42.317976 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.317951 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-xks-ca" pod "69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d" (UID: "69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:42.318078 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.317993 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-kube-api-access-s8lm9" pod "fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300" (UID: "fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300"). InnerVolumeSpecName "kube-api-access-s8lm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:42.318141 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.318074 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-kube-api-access-jj9mx" pod "69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d" (UID: "69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d"). InnerVolumeSpecName "kube-api-access-jj9mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:42.318323 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.318302 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-xks-ca" pod "fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300" (UID: "fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:42.418554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.418516 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:42.418554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.418556 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:42.418861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.418572 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8lm9\" (UniqueName: \"kubernetes.io/projected/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300-kube-api-access-s8lm9\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:42.418861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:42.418589 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj9mx\" (UniqueName: \"kubernetes.io/projected/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d-kube-api-access-jj9mx\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:43.006134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:43.006055 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" Oct 31 01:06:43.006134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:43.006091 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk" event={"ID":"fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300","Type":"ContainerDied","Data":"f9172f2fa3e8adaaf1d62efc48ce8a0d1451764c0ac8a02f183b4d7d91520e3b"} Oct 31 01:06:43.006328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:43.006180 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9172f2fa3e8adaaf1d62efc48ce8a0d1451764c0ac8a02f183b4d7d91520e3b" Oct 31 01:06:43.007572 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:43.007546 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" Oct 31 01:06:43.007710 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:43.007568 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr" event={"ID":"69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d","Type":"ContainerDied","Data":"5b392d57f50771416026a668c7a69b067caddab6054082f1c2cae5cad3cddb01"} Oct 31 01:06:43.007710 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:43.007603 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b392d57f50771416026a668c7a69b067caddab6054082f1c2cae5cad3cddb01" Oct 31 01:06:45.120483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.120446 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn"] Oct 31 01:06:45.121464 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.121438 5719 cpu_manager.go:414] "containerMap: removing container" podUID="fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300" containerName="revoke-keys" Oct 31 01:06:45.121464 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.121462 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300" containerName="revoke-keys" Oct 31 01:06:45.121616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.121484 5719 cpu_manager.go:414] "containerMap: removing container" podUID="69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d" containerName="revoke-keys" Oct 31 01:06:45.121616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.121491 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d" containerName="revoke-keys" Oct 31 01:06:45.121616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.121585 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300" containerName="revoke-keys" Oct 31 01:06:45.121616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.121597 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d" containerName="revoke-keys" Oct 31 01:06:45.125187 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.125170 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.167253 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.167229 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft"] Oct 31 01:06:45.171016 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.170998 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.221870 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.221845 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn"] Oct 31 01:06:45.248179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248158 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.248296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248190 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.248384 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248359 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7p9\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-kube-api-access-pp7p9\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.248470 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248448 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4aeae8-c3e2-4111-9931-afbd88a847db-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.248533 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248520 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.248590 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248573 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.248632 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248607 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.248747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248732 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.248818 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.248775 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.272319 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.272295 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft"] Oct 31 01:06:45.350047 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350014 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.350195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350086 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.350195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350124 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.350195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350149 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.350195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350184 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.350420 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350216 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.350420 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350354 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t642j\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-kube-api-access-t642j\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.350420 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350391 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7p9\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-kube-api-access-pp7p9\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.350582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350441 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914a635c-961a-4ecf-bc37-e67f25add9d8-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.350582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350446 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.350582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350511 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4aeae8-c3e2-4111-9931-afbd88a847db-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.350582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350548 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.350850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350615 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.350850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350658 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.350850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350676 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.350850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350773 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.350850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350810 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.351122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350872 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.351122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.350931 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.351241 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.351154 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.351241 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.351232 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4aeae8-c3e2-4111-9931-afbd88a847db-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.351367 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.351335 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.351575 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.351556 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.353333 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.353309 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.354021 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.354000 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.408872 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.408816 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.409124 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.409106 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7p9\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-kube-api-access-pp7p9\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.443760 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.443740 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:45.452021 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.451999 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t642j\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-kube-api-access-t642j\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452137 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452031 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914a635c-961a-4ecf-bc37-e67f25add9d8-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452137 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452071 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452137 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452096 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452135 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452224 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452276 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452314 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452544 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452362 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452544 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452502 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-data\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452719 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914a635c-961a-4ecf-bc37-e67f25add9d8-istiod-ca-cert\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452815 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452790 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-credential-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452872 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452807 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-certs\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.452872 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.452829 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-socket\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.454338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.454315 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-envoy\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.454684 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.454668 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-podinfo\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.512337 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.511734 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t642j\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-kube-api-access-t642j\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.512337 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.512183 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-token\") pod \"e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.783967 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.783896 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:45.878616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.878579 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn"] Oct 31 01:06:45.882788 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:45.882759 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4aeae8_c3e2_4111_9931_afbd88a847db.slice/crio-6782b2da9885d1e383e928e64c39c48874c9ccf4c8af7c1e6e8b784445eb1069 WatchSource:0}: Error finding container 6782b2da9885d1e383e928e64c39c48874c9ccf4c8af7c1e6e8b784445eb1069: Status 404 returned error can't find the container with id 6782b2da9885d1e383e928e64c39c48874c9ccf4c8af7c1e6e8b784445eb1069 Oct 31 01:06:45.885385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.885352 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:06:45.885466 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.885416 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:06:45.885466 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:45.885444 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:06:46.027543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.027514 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" event={"ID":"fe4aeae8-c3e2-4111-9931-afbd88a847db","Type":"ContainerStarted","Data":"6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e"} Oct 31 01:06:46.027786 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.027570 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" event={"ID":"fe4aeae8-c3e2-4111-9931-afbd88a847db","Type":"ContainerStarted","Data":"6782b2da9885d1e383e928e64c39c48874c9ccf4c8af7c1e6e8b784445eb1069"} Oct 31 01:06:46.146383 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.146319 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podStartSLOduration=1.146301995 podStartE2EDuration="1.146301995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:46.143213987 +0000 UTC m=+2287.679356294" watchObservedRunningTime="2026-10-31 01:06:46.146301995 +0000 UTC m=+2287.682444300" Oct 31 01:06:46.166376 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.166343 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft"] Oct 31 01:06:46.172294 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:46.172268 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914a635c_961a_4ecf_bc37_e67f25add9d8.slice/crio-1c506da83a2077e3485a8f76e45d38af3fb80f56de1f121ae414cab60d637c84 WatchSource:0}: Error finding container 1c506da83a2077e3485a8f76e45d38af3fb80f56de1f121ae414cab60d637c84: Status 404 returned error can't find the container with id 1c506da83a2077e3485a8f76e45d38af3fb80f56de1f121ae414cab60d637c84 Oct 31 01:06:46.175172 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.175140 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:06:46.175282 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.175214 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:06:46.175282 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.175258 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:06:46.444521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.444426 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:46.445806 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.445778 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 01:06:46.445909 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:46.445827 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 01:06:47.059213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:47.059172 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" event={"ID":"914a635c-961a-4ecf-bc37-e67f25add9d8","Type":"ContainerStarted","Data":"e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f"} Oct 31 01:06:47.059372 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:47.059239 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" event={"ID":"914a635c-961a-4ecf-bc37-e67f25add9d8","Type":"ContainerStarted","Data":"1c506da83a2077e3485a8f76e45d38af3fb80f56de1f121ae414cab60d637c84"} Oct 31 01:06:47.197429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:47.197369 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" podStartSLOduration=2.197356739 podStartE2EDuration="2.197356739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:47.196117498 +0000 UTC m=+2288.732259804" watchObservedRunningTime="2026-10-31 01:06:47.197356739 +0000 UTC m=+2288.733499044" Oct 31 01:06:47.448204 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:47.448164 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 01:06:47.448467 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:47.448439 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 01:06:47.784876 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:47.784796 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:47.786431 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:47.786407 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 01:06:47.786511 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:47.786453 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 01:06:48.448718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:48.447022 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 01:06:48.448718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:48.447079 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 01:06:48.784966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:48.784875 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 01:06:48.784966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:48.784932 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 01:06:49.448719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:49.445113 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 01:06:49.448719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:49.445177 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 01:06:49.784763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:49.784648 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 01:06:49.784763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:49.784738 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 01:06:50.445530 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:50.445078 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 01:06:50.445530 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:50.445152 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 01:06:50.784822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:50.784732 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 01:06:50.784822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:50.784794 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 01:06:51.448348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:51.448309 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 01:06:51.448642 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:51.448607 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 01:06:51.784669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:51.784585 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 01:06:51.784669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:51.784641 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 01:06:52.025840 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:52.025802 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:06:52.445167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:52.445137 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 31 01:06:52.445349 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:52.445204 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 31 01:06:52.785311 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:52.785225 5719 patch_prober.go:29] interesting pod/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 01:06:52.785311 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:52.785289 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 01:06:53.032368 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.032294 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv"] Oct 31 01:06:53.037402 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.037344 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.135802 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.135766 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:06:53.136254 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.136231 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:06:53.139920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.136460 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:06:53.140916 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.140892 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:06:53.141024 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.140938 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv"] Oct 31 01:06:53.184898 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.184870 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8fzx\" (UniqueName: \"kubernetes.io/projected/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-kube-api-access-b8fzx\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.185063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.185000 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.185063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.185041 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.286403 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.286363 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8fzx\" (UniqueName: \"kubernetes.io/projected/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-kube-api-access-b8fzx\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.286631 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.286449 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.286631 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.286475 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.287325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.287297 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.289200 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.289124 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.349299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.349223 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8fzx\" (UniqueName: \"kubernetes.io/projected/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-kube-api-access-b8fzx\") pod \"maas-api-revoke-sub-eed20ea9-gpbqv\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.357659 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.356874 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:53.814714 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.812661 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv"] Oct 31 01:06:53.839455 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:06:53.839406 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb354f7f8_d2da_48b3_88f5_4dfd40dbbcd5.slice/crio-23b2648cb9aebec2fa16b71ae0ec28ba63d1162609f78aa1a61066b986f1ce27 WatchSource:0}: Error finding container 23b2648cb9aebec2fa16b71ae0ec28ba63d1162609f78aa1a61066b986f1ce27: Status 404 returned error can't find the container with id 23b2648cb9aebec2fa16b71ae0ec28ba63d1162609f78aa1a61066b986f1ce27 Oct 31 01:06:53.961727 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.960235 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:53.961727 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.961036 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:53.961964 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.961745 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:06:53.991501 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.991474 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:53.992008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.991985 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:53.992807 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:53.992762 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:06:54.131872 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:54.131836 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" event={"ID":"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5","Type":"ContainerStarted","Data":"23b2648cb9aebec2fa16b71ae0ec28ba63d1162609f78aa1a61066b986f1ce27"} Oct 31 01:06:55.154876 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:55.154842 5719 generic.go:356] "Generic (PLEG): container finished" podID="b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5" containerID="d2282e4a59785e74a1b7a836fa4a7c21d211bd5ebc2712cbad9d2b351cdb16c3" exitCode=0 Oct 31 01:06:55.156639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:55.156614 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" event={"ID":"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5","Type":"ContainerDied","Data":"d2282e4a59785e74a1b7a836fa4a7c21d211bd5ebc2712cbad9d2b351cdb16c3"} Oct 31 01:06:56.306512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.306481 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:56.372686 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.372657 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-kube-api-access-b8fzx\" (UniqueName: \"kubernetes.io/projected/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-kube-api-access-b8fzx\") pod \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " Oct 31 01:06:56.372823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.372748 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-xks-ca\") pod \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " Oct 31 01:06:56.372823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.372777 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-openshift-ca\") pod \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\" (UID: \"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5\") " Oct 31 01:06:56.373179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.373153 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-openshift-ca" pod "b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5" (UID: "b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:06:56.374806 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.374783 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-kube-api-access-b8fzx" pod "b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5" (UID: "b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5"). InnerVolumeSpecName "kube-api-access-b8fzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:56.374901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.374831 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-xks-ca" pod "b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5" (UID: "b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:56.474085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.474020 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8fzx\" (UniqueName: \"kubernetes.io/projected/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-kube-api-access-b8fzx\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:56.474085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.474043 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:56.474085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.474053 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:06:56.915239 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.915206 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2"] Oct 31 01:06:56.918064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:56.918038 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-31ae40-g2pz2"] Oct 31 01:06:57.169201 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:57.169123 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" Oct 31 01:06:57.169353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:57.169127 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv" event={"ID":"b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5","Type":"ContainerDied","Data":"23b2648cb9aebec2fa16b71ae0ec28ba63d1162609f78aa1a61066b986f1ce27"} Oct 31 01:06:57.169353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:57.169243 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b2648cb9aebec2fa16b71ae0ec28ba63d1162609f78aa1a61066b986f1ce27" Oct 31 01:06:58.604374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:58.604345 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13840698-87de-4af9-882f-4bf6026ba9da" path="/var/lib/kubelet/pods/13840698-87de-4af9-882f-4bf6026ba9da/volumes" Oct 31 01:06:59.983525 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:59.983491 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw"] Oct 31 01:06:59.984018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:06:59.983759 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" containerID="cri-o://2e0e8fd7278a39215c6784621a46a364722c93bd055c8b99f7d03580ad33f481" gracePeriod=30 Oct 31 01:07:00.212186 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:00.212154 5719 patch_prober.go:29] interesting pod/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:07:00.212338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:00.212207 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:07:03.009772 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:07:03.009738 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:07:05.224478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.224441 5719 generic.go:356] "Generic (PLEG): container finished" podID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerID="2e0e8fd7278a39215c6784621a46a364722c93bd055c8b99f7d03580ad33f481" exitCode=0 Oct 31 01:07:05.224818 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.224508 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" event={"ID":"6818ade5-3ff1-4f90-a9f8-53acf2132d51","Type":"ContainerDied","Data":"2e0e8fd7278a39215c6784621a46a364722c93bd055c8b99f7d03580ad33f481"} Oct 31 01:07:05.244207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.244183 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 01:07:05.377519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377493 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-credential-socket\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.377722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377539 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-socket\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.377722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377571 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-data\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.377722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377623 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-certs\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.377722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377663 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istiod-ca-cert\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.377942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377777 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-envoy\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.377942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377807 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-credential-socket" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.377942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377908 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-token\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-token\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.377942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377926 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-data" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.377942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377938 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-podinfo\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.378188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377959 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-socket" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.378188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.377990 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-kube-api-access-kx9v7\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-kube-api-access-kx9v7\") pod \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\" (UID: \"6818ade5-3ff1-4f90-a9f8-53acf2132d51\") " Oct 31 01:07:05.378188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.378026 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-certs" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.378188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.378105 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istiod-ca-cert" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:05.378436 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.378415 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.378500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.378445 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.378500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.378460 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.378500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.378474 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.378500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.378489 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.380324 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.380300 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-token" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:05.380432 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.380335 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-kube-api-access-kx9v7" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "kube-api-access-kx9v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:05.380566 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.380544 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-podinfo" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:07:05.380921 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.380897 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-envoy" pod "6818ade5-3ff1-4f90-a9f8-53acf2132d51" (UID: "6818ade5-3ff1-4f90-a9f8-53acf2132d51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:07:05.479519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.479485 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.479519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.479515 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.479519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.479526 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6818ade5-3ff1-4f90-a9f8-53acf2132d51-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:05.479838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:05.479535 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx9v7\" (UniqueName: \"kubernetes.io/projected/6818ade5-3ff1-4f90-a9f8-53acf2132d51-kube-api-access-kx9v7\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:06.231915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:06.231884 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" Oct 31 01:07:06.232362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:06.231920 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw" event={"ID":"6818ade5-3ff1-4f90-a9f8-53acf2132d51","Type":"ContainerDied","Data":"2839764a0c0faea2373230d82d4ccb293a935185eadd8b1da0c6af65b522fc86"} Oct 31 01:07:06.232362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:06.231978 5719 scope.go:122] "RemoveContainer" containerID="2e0e8fd7278a39215c6784621a46a364722c93bd055c8b99f7d03580ad33f481" Oct 31 01:07:06.394490 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:06.394456 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw"] Oct 31 01:07:06.421520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:06.421491 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd98f0-openshift-default-55954fcc8f-qvtmw"] Oct 31 01:07:06.605358 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:06.605318 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" path="/var/lib/kubelet/pods/6818ade5-3ff1-4f90-a9f8-53acf2132d51/volumes" Oct 31 01:07:10.604173 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.604144 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst"] Oct 31 01:07:10.604754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.604735 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" Oct 31 01:07:10.604816 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.604755 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" Oct 31 01:07:10.604816 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.604782 5719 cpu_manager.go:414] "containerMap: removing container" podUID="b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5" containerName="revoke-keys" Oct 31 01:07:10.604816 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.604787 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5" containerName="revoke-keys" Oct 31 01:07:10.604911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.604867 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6818ade5-3ff1-4f90-a9f8-53acf2132d51" containerName="istio-proxy" Oct 31 01:07:10.604911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.604880 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5" containerName="revoke-keys" Oct 31 01:07:10.609807 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.609786 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.674362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.672597 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst"] Oct 31 01:07:10.701092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.701062 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:07:10.701374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.701106 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:07:10.717522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.717487 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:10.717522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.717512 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:07:10.744619 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.744591 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2t4s\" (UniqueName: \"kubernetes.io/projected/20143222-43f7-4b4e-b1a5-6c0c91038c17-kube-api-access-p2t4s\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.744750 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.744644 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.744817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.744801 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.845509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.845479 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.845636 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.845582 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2t4s\" (UniqueName: \"kubernetes.io/projected/20143222-43f7-4b4e-b1a5-6c0c91038c17-kube-api-access-p2t4s\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.845636 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.845617 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.846217 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.846198 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.847963 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.847947 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.904499 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.904442 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2t4s\" (UniqueName: \"kubernetes.io/projected/20143222-43f7-4b4e-b1a5-6c0c91038c17-kube-api-access-p2t4s\") pod \"maas-api-revoke-sub-e47d5cd5-xgmst\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:10.924164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:10.924139 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:11.302524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:11.302443 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst"] Oct 31 01:07:11.306899 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:07:11.306873 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20143222_43f7_4b4e_b1a5_6c0c91038c17.slice/crio-cae05f6c9681f65d21e29ec204b15e3b3f3b33c07ad724325178354ccbc149a8 WatchSource:0}: Error finding container cae05f6c9681f65d21e29ec204b15e3b3f3b33c07ad724325178354ccbc149a8: Status 404 returned error can't find the container with id cae05f6c9681f65d21e29ec204b15e3b3f3b33c07ad724325178354ccbc149a8 Oct 31 01:07:12.272442 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:12.272410 5719 generic.go:356] "Generic (PLEG): container finished" podID="20143222-43f7-4b4e-b1a5-6c0c91038c17" containerID="a9cc6ca624b32fbeba0800abae8f9858d170c15ac083da02b49c581b10696d46" exitCode=0 Oct 31 01:07:12.272833 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:12.272500 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" event={"ID":"20143222-43f7-4b4e-b1a5-6c0c91038c17","Type":"ContainerDied","Data":"a9cc6ca624b32fbeba0800abae8f9858d170c15ac083da02b49c581b10696d46"} Oct 31 01:07:12.272833 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:12.272553 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" event={"ID":"20143222-43f7-4b4e-b1a5-6c0c91038c17","Type":"ContainerStarted","Data":"cae05f6c9681f65d21e29ec204b15e3b3f3b33c07ad724325178354ccbc149a8"} Oct 31 01:07:13.425378 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.425355 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:13.578785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.578759 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-openshift-ca\") pod \"20143222-43f7-4b4e-b1a5-6c0c91038c17\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " Oct 31 01:07:13.578946 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.578893 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-xks-ca\") pod \"20143222-43f7-4b4e-b1a5-6c0c91038c17\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " Oct 31 01:07:13.579003 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.578978 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20143222-43f7-4b4e-b1a5-6c0c91038c17-kube-api-access-p2t4s\" (UniqueName: \"kubernetes.io/projected/20143222-43f7-4b4e-b1a5-6c0c91038c17-kube-api-access-p2t4s\") pod \"20143222-43f7-4b4e-b1a5-6c0c91038c17\" (UID: \"20143222-43f7-4b4e-b1a5-6c0c91038c17\") " Oct 31 01:07:13.579105 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.579080 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-openshift-ca" pod "20143222-43f7-4b4e-b1a5-6c0c91038c17" (UID: "20143222-43f7-4b4e-b1a5-6c0c91038c17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:13.579304 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.579289 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:13.581224 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.581200 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-xks-ca" pod "20143222-43f7-4b4e-b1a5-6c0c91038c17" (UID: "20143222-43f7-4b4e-b1a5-6c0c91038c17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:13.581320 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.581224 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20143222-43f7-4b4e-b1a5-6c0c91038c17-kube-api-access-p2t4s" pod "20143222-43f7-4b4e-b1a5-6c0c91038c17" (UID: "20143222-43f7-4b4e-b1a5-6c0c91038c17"). InnerVolumeSpecName "kube-api-access-p2t4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:13.680480 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.680450 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20143222-43f7-4b4e-b1a5-6c0c91038c17-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:13.680628 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:13.680485 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2t4s\" (UniqueName: \"kubernetes.io/projected/20143222-43f7-4b4e-b1a5-6c0c91038c17-kube-api-access-p2t4s\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:14.285213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:14.285186 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" Oct 31 01:07:14.285370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:14.285214 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst" event={"ID":"20143222-43f7-4b4e-b1a5-6c0c91038c17","Type":"ContainerDied","Data":"cae05f6c9681f65d21e29ec204b15e3b3f3b33c07ad724325178354ccbc149a8"} Oct 31 01:07:14.285370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:14.285257 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cae05f6c9681f65d21e29ec204b15e3b3f3b33c07ad724325178354ccbc149a8" Oct 31 01:07:21.839512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.839479 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9"] Oct 31 01:07:21.840056 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.840041 5719 cpu_manager.go:414] "containerMap: removing container" podUID="20143222-43f7-4b4e-b1a5-6c0c91038c17" containerName="revoke-keys" Oct 31 01:07:21.840108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.840058 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20143222-43f7-4b4e-b1a5-6c0c91038c17" containerName="revoke-keys" Oct 31 01:07:21.840172 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.840163 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="20143222-43f7-4b4e-b1a5-6c0c91038c17" containerName="revoke-keys" Oct 31 01:07:21.844454 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.844438 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:21.889240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.889208 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:21.889378 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.889242 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6x44\" (UniqueName: \"kubernetes.io/projected/81561031-f6f9-48f6-874b-ca096d06ae90-kube-api-access-h6x44\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:21.889378 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.889367 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:21.898600 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.898575 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9"] Oct 31 01:07:21.942635 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.942600 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:07:21.942823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.942680 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:07:21.942823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.942758 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:21.942823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.942775 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:07:21.989914 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.989887 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:21.990058 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.989918 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6x44\" (UniqueName: \"kubernetes.io/projected/81561031-f6f9-48f6-874b-ca096d06ae90-kube-api-access-h6x44\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:21.990058 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.989982 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:21.990665 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.990646 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:21.992233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:21.992214 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:22.049243 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:22.049209 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6x44\" (UniqueName: \"kubernetes.io/projected/81561031-f6f9-48f6-874b-ca096d06ae90-kube-api-access-h6x44\") pod \"maas-api-revoke-sub-99c9f0dd-hr5p9\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:22.158221 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:22.158139 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:22.531392 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:22.531302 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9"] Oct 31 01:07:22.535631 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:07:22.535601 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81561031_f6f9_48f6_874b_ca096d06ae90.slice/crio-925fe2d41db1b520191b6746301c51fd5428034b909ba319804d0fcf4f5f11f1 WatchSource:0}: Error finding container 925fe2d41db1b520191b6746301c51fd5428034b909ba319804d0fcf4f5f11f1: Status 404 returned error can't find the container with id 925fe2d41db1b520191b6746301c51fd5428034b909ba319804d0fcf4f5f11f1 Oct 31 01:07:23.364548 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:23.364512 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" event={"ID":"81561031-f6f9-48f6-874b-ca096d06ae90","Type":"ContainerStarted","Data":"ac7c3480ba52ffd30da7bd2d868dab7ea60357bf1ccefc543d45e167ce6b3987"} Oct 31 01:07:23.364917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:23.364579 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" event={"ID":"81561031-f6f9-48f6-874b-ca096d06ae90","Type":"ContainerStarted","Data":"925fe2d41db1b520191b6746301c51fd5428034b909ba319804d0fcf4f5f11f1"} Oct 31 01:07:23.476430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:23.476383 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" podStartSLOduration=2.476369713 podStartE2EDuration="2.476369713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:23.475855141 +0000 UTC m=+2325.011997446" watchObservedRunningTime="2026-10-31 01:07:23.476369713 +0000 UTC m=+2325.012512018" Oct 31 01:07:24.372646 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:24.372610 5719 generic.go:356] "Generic (PLEG): container finished" podID="81561031-f6f9-48f6-874b-ca096d06ae90" containerID="ac7c3480ba52ffd30da7bd2d868dab7ea60357bf1ccefc543d45e167ce6b3987" exitCode=0 Oct 31 01:07:24.373083 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:24.372725 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" event={"ID":"81561031-f6f9-48f6-874b-ca096d06ae90","Type":"ContainerDied","Data":"ac7c3480ba52ffd30da7bd2d868dab7ea60357bf1ccefc543d45e167ce6b3987"} Oct 31 01:07:25.522015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.521989 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:25.532400 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.532381 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-openshift-ca\") pod \"81561031-f6f9-48f6-874b-ca096d06ae90\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " Oct 31 01:07:25.532483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.532426 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81561031-f6f9-48f6-874b-ca096d06ae90-kube-api-access-h6x44\" (UniqueName: \"kubernetes.io/projected/81561031-f6f9-48f6-874b-ca096d06ae90-kube-api-access-h6x44\") pod \"81561031-f6f9-48f6-874b-ca096d06ae90\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " Oct 31 01:07:25.532483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.532461 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-xks-ca\") pod \"81561031-f6f9-48f6-874b-ca096d06ae90\" (UID: \"81561031-f6f9-48f6-874b-ca096d06ae90\") " Oct 31 01:07:25.532780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.532758 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-openshift-ca" pod "81561031-f6f9-48f6-874b-ca096d06ae90" (UID: "81561031-f6f9-48f6-874b-ca096d06ae90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:25.532939 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.532923 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:25.534564 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.534539 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81561031-f6f9-48f6-874b-ca096d06ae90-kube-api-access-h6x44" pod "81561031-f6f9-48f6-874b-ca096d06ae90" (UID: "81561031-f6f9-48f6-874b-ca096d06ae90"). InnerVolumeSpecName "kube-api-access-h6x44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:25.534842 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.534827 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-xks-ca" pod "81561031-f6f9-48f6-874b-ca096d06ae90" (UID: "81561031-f6f9-48f6-874b-ca096d06ae90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:25.634572 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.634499 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81561031-f6f9-48f6-874b-ca096d06ae90-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:25.634572 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:25.634535 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6x44\" (UniqueName: \"kubernetes.io/projected/81561031-f6f9-48f6-874b-ca096d06ae90-kube-api-access-h6x44\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:26.386609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:26.386579 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" Oct 31 01:07:26.386877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:26.386584 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9" event={"ID":"81561031-f6f9-48f6-874b-ca096d06ae90","Type":"ContainerDied","Data":"925fe2d41db1b520191b6746301c51fd5428034b909ba319804d0fcf4f5f11f1"} Oct 31 01:07:26.386877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:26.386729 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="925fe2d41db1b520191b6746301c51fd5428034b909ba319804d0fcf4f5f11f1" Oct 31 01:07:33.009423 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:07:33.009386 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:07:33.631742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:33.631686 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk"] Oct 31 01:07:33.678742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:33.678678 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4ab4aa-xvgxk"] Oct 31 01:07:34.603759 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:34.603725 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300" path="/var/lib/kubelet/pods/fbdb1ac5-ca09-40dd-a8b7-a5cff4cbc300/volumes" Oct 31 01:07:41.078248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:41.078219 5719 scope.go:122] "RemoveContainer" containerID="92d628500c53f7862870a6bdd5300884f0a92e5b446bcf15ebf7cf016e14f900" Oct 31 01:07:41.091301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:41.091271 5719 scope.go:122] "RemoveContainer" containerID="da454a574b2304c95146a1c8c40e10a8031b1568e2af9815f56fccf5ad04a696" Oct 31 01:07:41.123330 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:41.123279 5719 scope.go:122] "RemoveContainer" containerID="fdebb7ba6cc9d4100af9678fe651a3bcc4a2112e9fc8ec2e97da9505fccab4ca" Oct 31 01:07:46.229102 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.229026 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd"] Oct 31 01:07:46.229577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.229562 5719 cpu_manager.go:414] "containerMap: removing container" podUID="81561031-f6f9-48f6-874b-ca096d06ae90" containerName="revoke-keys" Oct 31 01:07:46.229624 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.229580 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81561031-f6f9-48f6-874b-ca096d06ae90" containerName="revoke-keys" Oct 31 01:07:46.229715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.229705 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="81561031-f6f9-48f6-874b-ca096d06ae90" containerName="revoke-keys" Oct 31 01:07:46.234704 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.234678 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.288811 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.288790 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd"] Oct 31 01:07:46.332319 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.332288 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:46.332446 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.332428 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:07:46.332936 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.332910 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:07:46.333015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.333001 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:07:46.347242 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.347214 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvt9\" (UniqueName: \"kubernetes.io/projected/a0204857-cb05-414f-9d9c-f122774df276-kube-api-access-zhvt9\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.347348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.347249 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0204857-cb05-414f-9d9c-f122774df276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.347450 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.347407 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0204857-cb05-414f-9d9c-f122774df276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.448832 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.448806 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvt9\" (UniqueName: \"kubernetes.io/projected/a0204857-cb05-414f-9d9c-f122774df276-kube-api-access-zhvt9\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.448934 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.448838 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0204857-cb05-414f-9d9c-f122774df276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.448934 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.448893 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0204857-cb05-414f-9d9c-f122774df276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.449522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.449500 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0204857-cb05-414f-9d9c-f122774df276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.451263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.451245 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0204857-cb05-414f-9d9c-f122774df276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.507861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.507814 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvt9\" (UniqueName: \"kubernetes.io/projected/a0204857-cb05-414f-9d9c-f122774df276-kube-api-access-zhvt9\") pod \"maas-api-revoke-sub-52eebf9e-2xnzd\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.547341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.547321 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:46.926515 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.926479 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd"] Oct 31 01:07:46.930946 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:07:46.930920 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0204857_cb05_414f_9d9c_f122774df276.slice/crio-23f98838bfd75dc4fede83b9e150eb3c7ae5cade307c9212ff2efaab13c480b9 WatchSource:0}: Error finding container 23f98838bfd75dc4fede83b9e150eb3c7ae5cade307c9212ff2efaab13c480b9: Status 404 returned error can't find the container with id 23f98838bfd75dc4fede83b9e150eb3c7ae5cade307c9212ff2efaab13c480b9 Oct 31 01:07:46.991865 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:46.991839 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:07:47.524340 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:47.524247 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" event={"ID":"a0204857-cb05-414f-9d9c-f122774df276","Type":"ContainerStarted","Data":"525affda30217642fe8abf39c9202be2d3f99335c8ba15611c8d972f01a4beb6"} Oct 31 01:07:47.524340 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:47.524320 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" event={"ID":"a0204857-cb05-414f-9d9c-f122774df276","Type":"ContainerStarted","Data":"23f98838bfd75dc4fede83b9e150eb3c7ae5cade307c9212ff2efaab13c480b9"} Oct 31 01:07:47.639976 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:47.639917 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" podStartSLOduration=1.639898606 podStartE2EDuration="1.639898606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:47.635879907 +0000 UTC m=+2349.172022214" watchObservedRunningTime="2026-10-31 01:07:47.639898606 +0000 UTC m=+2349.176040912" Oct 31 01:07:48.565719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:48.562672 5719 generic.go:356] "Generic (PLEG): container finished" podID="a0204857-cb05-414f-9d9c-f122774df276" containerID="525affda30217642fe8abf39c9202be2d3f99335c8ba15611c8d972f01a4beb6" exitCode=0 Oct 31 01:07:48.565719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:48.562799 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" event={"ID":"a0204857-cb05-414f-9d9c-f122774df276","Type":"ContainerDied","Data":"525affda30217642fe8abf39c9202be2d3f99335c8ba15611c8d972f01a4beb6"} Oct 31 01:07:49.717179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.717147 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:49.797244 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.797218 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0204857-cb05-414f-9d9c-f122774df276-kube-api-access-zhvt9\" (UniqueName: \"kubernetes.io/projected/a0204857-cb05-414f-9d9c-f122774df276-kube-api-access-zhvt9\") pod \"a0204857-cb05-414f-9d9c-f122774df276\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " Oct 31 01:07:49.797371 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.797247 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0204857-cb05-414f-9d9c-f122774df276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0204857-cb05-414f-9d9c-f122774df276-maas-api-xks-ca\") pod \"a0204857-cb05-414f-9d9c-f122774df276\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " Oct 31 01:07:49.797371 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.797292 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0204857-cb05-414f-9d9c-f122774df276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0204857-cb05-414f-9d9c-f122774df276-maas-api-openshift-ca\") pod \"a0204857-cb05-414f-9d9c-f122774df276\" (UID: \"a0204857-cb05-414f-9d9c-f122774df276\") " Oct 31 01:07:49.797679 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.797656 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0204857-cb05-414f-9d9c-f122774df276-maas-api-openshift-ca" pod "a0204857-cb05-414f-9d9c-f122774df276" (UID: "a0204857-cb05-414f-9d9c-f122774df276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:49.799300 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.799278 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0204857-cb05-414f-9d9c-f122774df276-kube-api-access-zhvt9" pod "a0204857-cb05-414f-9d9c-f122774df276" (UID: "a0204857-cb05-414f-9d9c-f122774df276"). InnerVolumeSpecName "kube-api-access-zhvt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:49.799383 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.799327 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0204857-cb05-414f-9d9c-f122774df276-maas-api-xks-ca" pod "a0204857-cb05-414f-9d9c-f122774df276" (UID: "a0204857-cb05-414f-9d9c-f122774df276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:49.898374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.898300 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvt9\" (UniqueName: \"kubernetes.io/projected/a0204857-cb05-414f-9d9c-f122774df276-kube-api-access-zhvt9\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:49.898374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.898327 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0204857-cb05-414f-9d9c-f122774df276-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:49.898374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:49.898337 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0204857-cb05-414f-9d9c-f122774df276-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:50.578204 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:50.578174 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" Oct 31 01:07:50.578380 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:50.578214 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd" event={"ID":"a0204857-cb05-414f-9d9c-f122774df276","Type":"ContainerDied","Data":"23f98838bfd75dc4fede83b9e150eb3c7ae5cade307c9212ff2efaab13c480b9"} Oct 31 01:07:50.578380 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:50.578283 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23f98838bfd75dc4fede83b9e150eb3c7ae5cade307c9212ff2efaab13c480b9" Oct 31 01:07:53.173649 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.173614 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl"] Oct 31 01:07:53.174158 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.174142 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a0204857-cb05-414f-9d9c-f122774df276" containerName="revoke-keys" Oct 31 01:07:53.174210 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.174160 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0204857-cb05-414f-9d9c-f122774df276" containerName="revoke-keys" Oct 31 01:07:53.174286 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.174277 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0204857-cb05-414f-9d9c-f122774df276" containerName="revoke-keys" Oct 31 01:07:53.177726 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.177708 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.228142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.228111 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:07:53.228142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.228123 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:07:53.275853 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.275814 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:07:53.276023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.275871 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:07:53.279075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.279051 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl"] Oct 31 01:07:53.332468 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.332436 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.332605 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.332571 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.332750 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.332735 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqvzv\" (UniqueName: \"kubernetes.io/projected/2f42a8e4-015d-478f-818b-95313d5cc9fe-kube-api-access-tqvzv\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.433456 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.433394 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvzv\" (UniqueName: \"kubernetes.io/projected/2f42a8e4-015d-478f-818b-95313d5cc9fe-kube-api-access-tqvzv\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.433456 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.433440 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.433648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.433506 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.434164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.434144 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.435964 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.435950 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.492534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.492508 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqvzv\" (UniqueName: \"kubernetes.io/projected/2f42a8e4-015d-478f-818b-95313d5cc9fe-kube-api-access-tqvzv\") pod \"maas-api-revoke-sub-28d5b65d-njqnl\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:53.792478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:53.792416 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:54.166513 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:54.166483 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl"] Oct 31 01:07:54.171020 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:07:54.170993 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f42a8e4_015d_478f_818b_95313d5cc9fe.slice/crio-4ea1f5d8eff63e5aa74eddfa2557327eb396fa8ce6909631ac949dd1020796f3 WatchSource:0}: Error finding container 4ea1f5d8eff63e5aa74eddfa2557327eb396fa8ce6909631ac949dd1020796f3: Status 404 returned error can't find the container with id 4ea1f5d8eff63e5aa74eddfa2557327eb396fa8ce6909631ac949dd1020796f3 Oct 31 01:07:54.608959 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:54.608930 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" event={"ID":"2f42a8e4-015d-478f-818b-95313d5cc9fe","Type":"ContainerStarted","Data":"d42282a9cef4a9e6fb35746851ec20871a8fcff0812bb1899565285a7a146394"} Oct 31 01:07:54.609338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:54.608983 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" event={"ID":"2f42a8e4-015d-478f-818b-95313d5cc9fe","Type":"ContainerStarted","Data":"4ea1f5d8eff63e5aa74eddfa2557327eb396fa8ce6909631ac949dd1020796f3"} Oct 31 01:07:54.723563 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:54.723482 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" podStartSLOduration=1.723468923 podStartE2EDuration="1.723468923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:54.720612074 +0000 UTC m=+2356.256754379" watchObservedRunningTime="2026-10-31 01:07:54.723468923 +0000 UTC m=+2356.259611269" Oct 31 01:07:55.617055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:55.617023 5719 generic.go:356] "Generic (PLEG): container finished" podID="2f42a8e4-015d-478f-818b-95313d5cc9fe" containerID="d42282a9cef4a9e6fb35746851ec20871a8fcff0812bb1899565285a7a146394" exitCode=0 Oct 31 01:07:55.617533 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:55.617116 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" event={"ID":"2f42a8e4-015d-478f-818b-95313d5cc9fe","Type":"ContainerDied","Data":"d42282a9cef4a9e6fb35746851ec20871a8fcff0812bb1899565285a7a146394"} Oct 31 01:07:56.769374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.769351 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:56.868733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.868682 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f42a8e4-015d-478f-818b-95313d5cc9fe-kube-api-access-tqvzv\" (UniqueName: \"kubernetes.io/projected/2f42a8e4-015d-478f-818b-95313d5cc9fe-kube-api-access-tqvzv\") pod \"2f42a8e4-015d-478f-818b-95313d5cc9fe\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " Oct 31 01:07:56.868851 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.868747 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-openshift-ca\") pod \"2f42a8e4-015d-478f-818b-95313d5cc9fe\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " Oct 31 01:07:56.868851 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.868804 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-xks-ca\") pod \"2f42a8e4-015d-478f-818b-95313d5cc9fe\" (UID: \"2f42a8e4-015d-478f-818b-95313d5cc9fe\") " Oct 31 01:07:56.869088 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.869065 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-openshift-ca" pod "2f42a8e4-015d-478f-818b-95313d5cc9fe" (UID: "2f42a8e4-015d-478f-818b-95313d5cc9fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:07:56.870803 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.870781 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f42a8e4-015d-478f-818b-95313d5cc9fe-kube-api-access-tqvzv" pod "2f42a8e4-015d-478f-818b-95313d5cc9fe" (UID: "2f42a8e4-015d-478f-818b-95313d5cc9fe"). InnerVolumeSpecName "kube-api-access-tqvzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:07:56.870921 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.870901 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-xks-ca" pod "2f42a8e4-015d-478f-818b-95313d5cc9fe" (UID: "2f42a8e4-015d-478f-818b-95313d5cc9fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:07:56.970266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.970233 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqvzv\" (UniqueName: \"kubernetes.io/projected/2f42a8e4-015d-478f-818b-95313d5cc9fe-kube-api-access-tqvzv\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:56.970266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.970258 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:56.970266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:56.970271 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f42a8e4-015d-478f-818b-95313d5cc9fe-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:07:57.631583 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:57.631553 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" Oct 31 01:07:57.631743 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:57.631556 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl" event={"ID":"2f42a8e4-015d-478f-818b-95313d5cc9fe","Type":"ContainerDied","Data":"4ea1f5d8eff63e5aa74eddfa2557327eb396fa8ce6909631ac949dd1020796f3"} Oct 31 01:07:57.631743 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:07:57.631677 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ea1f5d8eff63e5aa74eddfa2557327eb396fa8ce6909631ac949dd1020796f3" Oct 31 01:08:01.732291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.732257 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96"] Oct 31 01:08:01.732765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.732750 5719 cpu_manager.go:414] "containerMap: removing container" podUID="2f42a8e4-015d-478f-818b-95313d5cc9fe" containerName="revoke-keys" Oct 31 01:08:01.732765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.732765 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f42a8e4-015d-478f-818b-95313d5cc9fe" containerName="revoke-keys" Oct 31 01:08:01.732860 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.732853 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f42a8e4-015d-478f-818b-95313d5cc9fe" containerName="revoke-keys" Oct 31 01:08:01.736108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.736093 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.821577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.821547 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.821577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.821578 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.821802 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.821613 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5ld\" (UniqueName: \"kubernetes.io/projected/5826ee17-9007-4e30-a02c-9701137c8757-kube-api-access-8k5ld\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.840932 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.840909 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:01.841046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.840966 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:08:01.841046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.840967 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:01.841165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.841047 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:01.844011 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.843989 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96"] Oct 31 01:08:01.922250 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.922224 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.922351 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.922260 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.922399 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.922378 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5ld\" (UniqueName: \"kubernetes.io/projected/5826ee17-9007-4e30-a02c-9701137c8757-kube-api-access-8k5ld\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.922879 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.922862 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.924664 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.924646 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:01.981731 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:01.981680 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5ld\" (UniqueName: \"kubernetes.io/projected/5826ee17-9007-4e30-a02c-9701137c8757-kube-api-access-8k5ld\") pod \"maas-api-revoke-sub-03dd63dc-xtv96\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:02.048733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.048674 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:02.423096 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.423063 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96"] Oct 31 01:08:02.427422 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:02.427398 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5826ee17_9007_4e30_a02c_9701137c8757.slice/crio-93feb0e491528b2fdf8a4e9df691ba7d64d78fbc86f48a2ca6d135bb9fbd4b64 WatchSource:0}: Error finding container 93feb0e491528b2fdf8a4e9df691ba7d64d78fbc86f48a2ca6d135bb9fbd4b64: Status 404 returned error can't find the container with id 93feb0e491528b2fdf8a4e9df691ba7d64d78fbc86f48a2ca6d135bb9fbd4b64 Oct 31 01:08:02.584212 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.584186 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8"] Oct 31 01:08:02.589257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.589237 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.629280 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.629252 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn8m\" (UniqueName: \"kubernetes.io/projected/756baf43-313a-4756-a143-0562bdafec50-kube-api-access-ljn8m\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.629413 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.629395 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756baf43-313a-4756-a143-0562bdafec50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.629539 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.629520 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756baf43-313a-4756-a143-0562bdafec50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.644081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.644051 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8"] Oct 31 01:08:02.663153 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.663124 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" event={"ID":"5826ee17-9007-4e30-a02c-9701137c8757","Type":"ContainerStarted","Data":"93feb0e491528b2fdf8a4e9df691ba7d64d78fbc86f48a2ca6d135bb9fbd4b64"} Oct 31 01:08:02.730650 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.730629 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756baf43-313a-4756-a143-0562bdafec50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.730762 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.730672 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn8m\" (UniqueName: \"kubernetes.io/projected/756baf43-313a-4756-a143-0562bdafec50-kube-api-access-ljn8m\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.730811 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.730769 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756baf43-313a-4756-a143-0562bdafec50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.731386 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.731340 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756baf43-313a-4756-a143-0562bdafec50-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.732839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.732817 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756baf43-313a-4756-a143-0562bdafec50-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.789862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.789836 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn8m\" (UniqueName: \"kubernetes.io/projected/756baf43-313a-4756-a143-0562bdafec50-kube-api-access-ljn8m\") pod \"maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:02.905100 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:02.905074 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:03.013295 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:08:03.013261 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:08:03.281070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:03.280987 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8"] Oct 31 01:08:03.285624 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:03.285601 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756baf43_313a_4756_a143_0562bdafec50.slice/crio-13ca15e7082c3551074d57245c2996f47c2af5cf2257e562dfb78849bed2509a WatchSource:0}: Error finding container 13ca15e7082c3551074d57245c2996f47c2af5cf2257e562dfb78849bed2509a: Status 404 returned error can't find the container with id 13ca15e7082c3551074d57245c2996f47c2af5cf2257e562dfb78849bed2509a Oct 31 01:08:03.673893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:03.673859 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" event={"ID":"756baf43-313a-4756-a143-0562bdafec50","Type":"ContainerStarted","Data":"beee0b47520dda070cb8b957fe34942c93f9007d2db2a21958dc3fde5a5e933b"} Oct 31 01:08:03.674029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:03.673951 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" event={"ID":"756baf43-313a-4756-a143-0562bdafec50","Type":"ContainerStarted","Data":"13ca15e7082c3551074d57245c2996f47c2af5cf2257e562dfb78849bed2509a"} Oct 31 01:08:03.675532 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:03.675505 5719 generic.go:356] "Generic (PLEG): container finished" podID="5826ee17-9007-4e30-a02c-9701137c8757" containerID="f9fe7f2a1b3161dac430206eb337961e38316e20bc805c6ce8eeb542ab71258d" exitCode=0 Oct 31 01:08:03.675659 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:03.675589 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" event={"ID":"5826ee17-9007-4e30-a02c-9701137c8757","Type":"ContainerDied","Data":"f9fe7f2a1b3161dac430206eb337961e38316e20bc805c6ce8eeb542ab71258d"} Oct 31 01:08:03.786969 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:03.786930 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" podStartSLOduration=1.786918599 podStartE2EDuration="1.786918599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:03.785635159 +0000 UTC m=+2365.321777466" watchObservedRunningTime="2026-10-31 01:08:03.786918599 +0000 UTC m=+2365.323060926" Oct 31 01:08:04.683000 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.682969 5719 generic.go:356] "Generic (PLEG): container finished" podID="756baf43-313a-4756-a143-0562bdafec50" containerID="beee0b47520dda070cb8b957fe34942c93f9007d2db2a21958dc3fde5a5e933b" exitCode=0 Oct 31 01:08:04.683163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.683054 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" event={"ID":"756baf43-313a-4756-a143-0562bdafec50","Type":"ContainerDied","Data":"beee0b47520dda070cb8b957fe34942c93f9007d2db2a21958dc3fde5a5e933b"} Oct 31 01:08:04.827485 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.827459 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:04.852806 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.852780 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-openshift-ca\") pod \"5826ee17-9007-4e30-a02c-9701137c8757\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " Oct 31 01:08:04.852951 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.852833 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-xks-ca\") pod \"5826ee17-9007-4e30-a02c-9701137c8757\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " Oct 31 01:08:04.853038 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.853019 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5826ee17-9007-4e30-a02c-9701137c8757-kube-api-access-8k5ld\" (UniqueName: \"kubernetes.io/projected/5826ee17-9007-4e30-a02c-9701137c8757-kube-api-access-8k5ld\") pod \"5826ee17-9007-4e30-a02c-9701137c8757\" (UID: \"5826ee17-9007-4e30-a02c-9701137c8757\") " Oct 31 01:08:04.853173 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.853153 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-openshift-ca" pod "5826ee17-9007-4e30-a02c-9701137c8757" (UID: "5826ee17-9007-4e30-a02c-9701137c8757"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:04.853501 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.853449 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:04.855900 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.855870 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-xks-ca" pod "5826ee17-9007-4e30-a02c-9701137c8757" (UID: "5826ee17-9007-4e30-a02c-9701137c8757"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:04.856681 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.856658 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5826ee17-9007-4e30-a02c-9701137c8757-kube-api-access-8k5ld" pod "5826ee17-9007-4e30-a02c-9701137c8757" (UID: "5826ee17-9007-4e30-a02c-9701137c8757"). InnerVolumeSpecName "kube-api-access-8k5ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:04.954259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.954183 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k5ld\" (UniqueName: \"kubernetes.io/projected/5826ee17-9007-4e30-a02c-9701137c8757-kube-api-access-8k5ld\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:04.954259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:04.954216 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5826ee17-9007-4e30-a02c-9701137c8757-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:05.690704 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.690664 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" Oct 31 01:08:05.690864 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.690734 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96" event={"ID":"5826ee17-9007-4e30-a02c-9701137c8757","Type":"ContainerDied","Data":"93feb0e491528b2fdf8a4e9df691ba7d64d78fbc86f48a2ca6d135bb9fbd4b64"} Oct 31 01:08:05.690864 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.690780 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93feb0e491528b2fdf8a4e9df691ba7d64d78fbc86f48a2ca6d135bb9fbd4b64" Oct 31 01:08:05.839272 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.839249 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:05.861020 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.860997 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756baf43-313a-4756-a143-0562bdafec50-kube-api-access-ljn8m\" (UniqueName: \"kubernetes.io/projected/756baf43-313a-4756-a143-0562bdafec50-kube-api-access-ljn8m\") pod \"756baf43-313a-4756-a143-0562bdafec50\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " Oct 31 01:08:05.861111 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.861033 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756baf43-313a-4756-a143-0562bdafec50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756baf43-313a-4756-a143-0562bdafec50-maas-api-openshift-ca\") pod \"756baf43-313a-4756-a143-0562bdafec50\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " Oct 31 01:08:05.861159 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.861137 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/756baf43-313a-4756-a143-0562bdafec50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756baf43-313a-4756-a143-0562bdafec50-maas-api-xks-ca\") pod \"756baf43-313a-4756-a143-0562bdafec50\" (UID: \"756baf43-313a-4756-a143-0562bdafec50\") " Oct 31 01:08:05.861461 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.861431 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756baf43-313a-4756-a143-0562bdafec50-maas-api-openshift-ca" pod "756baf43-313a-4756-a143-0562bdafec50" (UID: "756baf43-313a-4756-a143-0562bdafec50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:05.863181 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.863163 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/756baf43-313a-4756-a143-0562bdafec50-maas-api-xks-ca" pod "756baf43-313a-4756-a143-0562bdafec50" (UID: "756baf43-313a-4756-a143-0562bdafec50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:05.863258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.863194 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756baf43-313a-4756-a143-0562bdafec50-kube-api-access-ljn8m" pod "756baf43-313a-4756-a143-0562bdafec50" (UID: "756baf43-313a-4756-a143-0562bdafec50"). InnerVolumeSpecName "kube-api-access-ljn8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:05.962576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.962513 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljn8m\" (UniqueName: \"kubernetes.io/projected/756baf43-313a-4756-a143-0562bdafec50-kube-api-access-ljn8m\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:05.962576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.962549 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/756baf43-313a-4756-a143-0562bdafec50-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:05.962576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:05.962567 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/756baf43-313a-4756-a143-0562bdafec50-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:06.698419 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:06.698389 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" Oct 31 01:08:06.698558 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:06.698428 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8" event={"ID":"756baf43-313a-4756-a143-0562bdafec50","Type":"ContainerDied","Data":"13ca15e7082c3551074d57245c2996f47c2af5cf2257e562dfb78849bed2509a"} Oct 31 01:08:06.698558 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:06.698478 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ca15e7082c3551074d57245c2996f47c2af5cf2257e562dfb78849bed2509a" Oct 31 01:08:09.620776 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.620746 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57"] Oct 31 01:08:09.621241 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.621226 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5826ee17-9007-4e30-a02c-9701137c8757" containerName="revoke-keys" Oct 31 01:08:09.621289 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.621243 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5826ee17-9007-4e30-a02c-9701137c8757" containerName="revoke-keys" Oct 31 01:08:09.621289 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.621274 5719 cpu_manager.go:414] "containerMap: removing container" podUID="756baf43-313a-4756-a143-0562bdafec50" containerName="revoke-keys" Oct 31 01:08:09.621289 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.621278 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756baf43-313a-4756-a143-0562bdafec50" containerName="revoke-keys" Oct 31 01:08:09.621429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.621348 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="756baf43-313a-4756-a143-0562bdafec50" containerName="revoke-keys" Oct 31 01:08:09.621429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.621362 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5826ee17-9007-4e30-a02c-9701137c8757" containerName="revoke-keys" Oct 31 01:08:09.624926 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.624909 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.632329 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.632024 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjncs\" (UniqueName: \"kubernetes.io/projected/64053c11-c2fd-4b5d-9728-7a6647a555cf-kube-api-access-hjncs\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.632329 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.632091 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.632329 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.632206 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.676787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.676758 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:09.678557 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.678533 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:09.680457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.680435 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57"] Oct 31 01:08:09.724338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.724315 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:09.724450 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.724364 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:08:09.733434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.733414 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.733552 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.733460 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.733618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.733582 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjncs\" (UniqueName: \"kubernetes.io/projected/64053c11-c2fd-4b5d-9728-7a6647a555cf-kube-api-access-hjncs\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.734128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.734111 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.735946 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.735929 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.792165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.792145 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjncs\" (UniqueName: \"kubernetes.io/projected/64053c11-c2fd-4b5d-9728-7a6647a555cf-kube-api-access-hjncs\") pod \"maas-api-revoke-sub-e200a174-lxm57\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:09.938448 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:09.938394 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:10.311532 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:10.311418 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57"] Oct 31 01:08:10.315842 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:10.315808 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64053c11_c2fd_4b5d_9728_7a6647a555cf.slice/crio-d0988980d66e3c7741796c01e70ef9c4ed7c1e78eaa20c6e159051ad57391b93 WatchSource:0}: Error finding container d0988980d66e3c7741796c01e70ef9c4ed7c1e78eaa20c6e159051ad57391b93: Status 404 returned error can't find the container with id d0988980d66e3c7741796c01e70ef9c4ed7c1e78eaa20c6e159051ad57391b93 Oct 31 01:08:10.749225 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:10.749194 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" event={"ID":"64053c11-c2fd-4b5d-9728-7a6647a555cf","Type":"ContainerStarted","Data":"dbb9bd2cbfae98f7cada1a959fba8bbd697a09db19d4447def3d20e772c368a5"} Oct 31 01:08:10.749560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:10.749254 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" event={"ID":"64053c11-c2fd-4b5d-9728-7a6647a555cf","Type":"ContainerStarted","Data":"d0988980d66e3c7741796c01e70ef9c4ed7c1e78eaa20c6e159051ad57391b93"} Oct 31 01:08:10.861134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:10.861059 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" podStartSLOduration=1.861045409 podStartE2EDuration="1.861045409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:10.860047135 +0000 UTC m=+2372.396189450" watchObservedRunningTime="2026-10-31 01:08:10.861045409 +0000 UTC m=+2372.397187714" Oct 31 01:08:11.760584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:11.760542 5719 generic.go:356] "Generic (PLEG): container finished" podID="64053c11-c2fd-4b5d-9728-7a6647a555cf" containerID="dbb9bd2cbfae98f7cada1a959fba8bbd697a09db19d4447def3d20e772c368a5" exitCode=0 Oct 31 01:08:11.761075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:11.760631 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" event={"ID":"64053c11-c2fd-4b5d-9728-7a6647a555cf","Type":"ContainerDied","Data":"dbb9bd2cbfae98f7cada1a959fba8bbd697a09db19d4447def3d20e772c368a5"} Oct 31 01:08:12.914224 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:12.914203 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:12.967759 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:12.967730 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-openshift-ca\") pod \"64053c11-c2fd-4b5d-9728-7a6647a555cf\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " Oct 31 01:08:12.967897 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:12.967788 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64053c11-c2fd-4b5d-9728-7a6647a555cf-kube-api-access-hjncs\" (UniqueName: \"kubernetes.io/projected/64053c11-c2fd-4b5d-9728-7a6647a555cf-kube-api-access-hjncs\") pod \"64053c11-c2fd-4b5d-9728-7a6647a555cf\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " Oct 31 01:08:12.967978 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:12.967954 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-xks-ca\") pod \"64053c11-c2fd-4b5d-9728-7a6647a555cf\" (UID: \"64053c11-c2fd-4b5d-9728-7a6647a555cf\") " Oct 31 01:08:12.968139 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:12.968112 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-openshift-ca" pod "64053c11-c2fd-4b5d-9728-7a6647a555cf" (UID: "64053c11-c2fd-4b5d-9728-7a6647a555cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:12.968257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:12.968242 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:12.970051 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:12.970028 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-xks-ca" pod "64053c11-c2fd-4b5d-9728-7a6647a555cf" (UID: "64053c11-c2fd-4b5d-9728-7a6647a555cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:12.970128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:12.970030 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64053c11-c2fd-4b5d-9728-7a6647a555cf-kube-api-access-hjncs" pod "64053c11-c2fd-4b5d-9728-7a6647a555cf" (UID: "64053c11-c2fd-4b5d-9728-7a6647a555cf"). InnerVolumeSpecName "kube-api-access-hjncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:13.069240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:13.069204 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjncs\" (UniqueName: \"kubernetes.io/projected/64053c11-c2fd-4b5d-9728-7a6647a555cf-kube-api-access-hjncs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:13.069240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:13.069233 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64053c11-c2fd-4b5d-9728-7a6647a555cf-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:13.773764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:13.773721 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" event={"ID":"64053c11-c2fd-4b5d-9728-7a6647a555cf","Type":"ContainerDied","Data":"d0988980d66e3c7741796c01e70ef9c4ed7c1e78eaa20c6e159051ad57391b93"} Oct 31 01:08:13.773764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:13.773751 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57" Oct 31 01:08:13.773991 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:13.773776 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0988980d66e3c7741796c01e70ef9c4ed7c1e78eaa20c6e159051ad57391b93" Oct 31 01:08:17.031209 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.031179 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb"] Oct 31 01:08:17.031666 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.031653 5719 cpu_manager.go:414] "containerMap: removing container" podUID="64053c11-c2fd-4b5d-9728-7a6647a555cf" containerName="revoke-keys" Oct 31 01:08:17.031740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.031667 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64053c11-c2fd-4b5d-9728-7a6647a555cf" containerName="revoke-keys" Oct 31 01:08:17.031796 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.031786 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="64053c11-c2fd-4b5d-9728-7a6647a555cf" containerName="revoke-keys" Oct 31 01:08:17.035184 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.035168 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.110791 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.110754 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.111037 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.111020 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.111250 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.111234 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz69l\" (UniqueName: \"kubernetes.io/projected/43de3169-70bf-4f38-9c9d-9ea88eda15af-kube-api-access-nz69l\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.134465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.134435 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:17.134465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.134435 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:17.134663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.134496 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:17.134663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.134571 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:08:17.137753 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.137731 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb"] Oct 31 01:08:17.212366 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.212339 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.212465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.212445 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz69l\" (UniqueName: \"kubernetes.io/projected/43de3169-70bf-4f38-9c9d-9ea88eda15af-kube-api-access-nz69l\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.212581 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.212565 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.213197 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.213174 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.214784 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.214765 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.271443 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.271399 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz69l\" (UniqueName: \"kubernetes.io/projected/43de3169-70bf-4f38-9c9d-9ea88eda15af-kube-api-access-nz69l\") pod \"maas-api-revoke-sub-c44aa87d-h2vlb\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.351342 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.351320 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:17.726400 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.726324 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb"] Oct 31 01:08:17.730991 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:17.730964 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43de3169_70bf_4f38_9c9d_9ea88eda15af.slice/crio-de527cf56202a0d8bd7e89e8dfd8dffa1519586cf2d534e46d62c4cef00e1902 WatchSource:0}: Error finding container de527cf56202a0d8bd7e89e8dfd8dffa1519586cf2d534e46d62c4cef00e1902: Status 404 returned error can't find the container with id de527cf56202a0d8bd7e89e8dfd8dffa1519586cf2d534e46d62c4cef00e1902 Oct 31 01:08:17.801184 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:17.801158 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" event={"ID":"43de3169-70bf-4f38-9c9d-9ea88eda15af","Type":"ContainerStarted","Data":"de527cf56202a0d8bd7e89e8dfd8dffa1519586cf2d534e46d62c4cef00e1902"} Oct 31 01:08:18.818715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:18.816200 5719 generic.go:356] "Generic (PLEG): container finished" podID="43de3169-70bf-4f38-9c9d-9ea88eda15af" containerID="71e54d4ba03abda88b88f9b46db043dd327e8a0740443b1b4ce4b8e348043703" exitCode=0 Oct 31 01:08:18.818715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:18.816389 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" event={"ID":"43de3169-70bf-4f38-9c9d-9ea88eda15af","Type":"ContainerDied","Data":"71e54d4ba03abda88b88f9b46db043dd327e8a0740443b1b4ce4b8e348043703"} Oct 31 01:08:19.963891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:19.963869 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:20.045235 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.045205 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43de3169-70bf-4f38-9c9d-9ea88eda15af-kube-api-access-nz69l\" (UniqueName: \"kubernetes.io/projected/43de3169-70bf-4f38-9c9d-9ea88eda15af-kube-api-access-nz69l\") pod \"43de3169-70bf-4f38-9c9d-9ea88eda15af\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " Oct 31 01:08:20.045415 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.045354 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-openshift-ca\") pod \"43de3169-70bf-4f38-9c9d-9ea88eda15af\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " Oct 31 01:08:20.045415 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.045377 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-xks-ca\") pod \"43de3169-70bf-4f38-9c9d-9ea88eda15af\" (UID: \"43de3169-70bf-4f38-9c9d-9ea88eda15af\") " Oct 31 01:08:20.045800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.045767 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-openshift-ca" pod "43de3169-70bf-4f38-9c9d-9ea88eda15af" (UID: "43de3169-70bf-4f38-9c9d-9ea88eda15af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:20.048002 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.047977 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-xks-ca" pod "43de3169-70bf-4f38-9c9d-9ea88eda15af" (UID: "43de3169-70bf-4f38-9c9d-9ea88eda15af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:20.048804 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.048783 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43de3169-70bf-4f38-9c9d-9ea88eda15af-kube-api-access-nz69l" pod "43de3169-70bf-4f38-9c9d-9ea88eda15af" (UID: "43de3169-70bf-4f38-9c9d-9ea88eda15af"). InnerVolumeSpecName "kube-api-access-nz69l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:20.147086 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.147016 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz69l\" (UniqueName: \"kubernetes.io/projected/43de3169-70bf-4f38-9c9d-9ea88eda15af-kube-api-access-nz69l\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:20.147086 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.147048 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:20.147086 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.147067 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43de3169-70bf-4f38-9c9d-9ea88eda15af-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:20.618283 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.618235 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc"] Oct 31 01:08:20.621029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.621005 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8c08e7-22dxc"] Oct 31 01:08:20.830117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.830090 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" Oct 31 01:08:20.830299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.830127 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb" event={"ID":"43de3169-70bf-4f38-9c9d-9ea88eda15af","Type":"ContainerDied","Data":"de527cf56202a0d8bd7e89e8dfd8dffa1519586cf2d534e46d62c4cef00e1902"} Oct 31 01:08:20.830299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:20.830204 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de527cf56202a0d8bd7e89e8dfd8dffa1519586cf2d534e46d62c4cef00e1902" Oct 31 01:08:22.603453 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:22.603423 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a66c4055-a8e5-43ee-ad5a-41a14a7222ee" path="/var/lib/kubelet/pods/a66c4055-a8e5-43ee-ad5a-41a14a7222ee/volumes" Oct 31 01:08:31.027832 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:31.027795 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:08:31.638276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:31.638245 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8"] Oct 31 01:08:31.640968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:31.640947 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-add5bf-gnss8"] Oct 31 01:08:32.437304 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.437262 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5"] Oct 31 01:08:32.437823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.437808 5719 cpu_manager.go:414] "containerMap: removing container" podUID="43de3169-70bf-4f38-9c9d-9ea88eda15af" containerName="revoke-keys" Oct 31 01:08:32.437880 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.437824 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43de3169-70bf-4f38-9c9d-9ea88eda15af" containerName="revoke-keys" Oct 31 01:08:32.437916 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.437911 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="43de3169-70bf-4f38-9c9d-9ea88eda15af" containerName="revoke-keys" Oct 31 01:08:32.441256 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.441240 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.493758 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.493339 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:32.493758 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.493574 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:32.494008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.493809 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:32.541141 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.541115 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:08:32.546153 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.546129 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5"] Oct 31 01:08:32.571413 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.571393 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.571515 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.571423 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvnm\" (UniqueName: \"kubernetes.io/projected/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-kube-api-access-7bvnm\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.571515 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.571464 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.606352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.606324 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756baf43-313a-4756-a143-0562bdafec50" path="/var/lib/kubelet/pods/756baf43-313a-4756-a143-0562bdafec50/volumes" Oct 31 01:08:32.672001 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.671969 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.672142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.672012 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvnm\" (UniqueName: \"kubernetes.io/projected/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-kube-api-access-7bvnm\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.672142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.672055 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.672861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.672842 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.674455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.674434 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.731621 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.731554 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvnm\" (UniqueName: \"kubernetes.io/projected/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-kube-api-access-7bvnm\") pod \"maas-api-revoke-sub-007bea65-twns5\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:32.754734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:32.754707 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:33.014125 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:08:33.014025 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:08:33.142738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:33.142685 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5"] Oct 31 01:08:33.147668 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:33.147636 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2bcb6f_d7da_4e7e_92b2_9817ab886d88.slice/crio-d189fc6314011d118aa3d2b2cfc442bca4f1d11886ad43dfde29f6c59ffde560 WatchSource:0}: Error finding container d189fc6314011d118aa3d2b2cfc442bca4f1d11886ad43dfde29f6c59ffde560: Status 404 returned error can't find the container with id d189fc6314011d118aa3d2b2cfc442bca4f1d11886ad43dfde29f6c59ffde560 Oct 31 01:08:33.930155 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:33.930122 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" event={"ID":"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88","Type":"ContainerStarted","Data":"a773f5df1b53b5e03c88395ff72df561a190077312e55f97557c241931e33896"} Oct 31 01:08:33.930579 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:33.930181 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" event={"ID":"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88","Type":"ContainerStarted","Data":"d189fc6314011d118aa3d2b2cfc442bca4f1d11886ad43dfde29f6c59ffde560"} Oct 31 01:08:34.044941 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:34.044891 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" podStartSLOduration=2.044876314 podStartE2EDuration="2.044876314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:34.041092439 +0000 UTC m=+2395.577234745" watchObservedRunningTime="2026-10-31 01:08:34.044876314 +0000 UTC m=+2395.581018618" Oct 31 01:08:34.937968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:34.937932 5719 generic.go:356] "Generic (PLEG): container finished" podID="7b2bcb6f-d7da-4e7e-92b2-9817ab886d88" containerID="a773f5df1b53b5e03c88395ff72df561a190077312e55f97557c241931e33896" exitCode=0 Oct 31 01:08:34.938371 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:34.938017 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" event={"ID":"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88","Type":"ContainerDied","Data":"a773f5df1b53b5e03c88395ff72df561a190077312e55f97557c241931e33896"} Oct 31 01:08:36.095359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.095328 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:36.211355 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.211285 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-xks-ca\") pod \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " Oct 31 01:08:36.211547 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.211360 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-openshift-ca\") pod \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " Oct 31 01:08:36.211547 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.211522 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-kube-api-access-7bvnm\" (UniqueName: \"kubernetes.io/projected/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-kube-api-access-7bvnm\") pod \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\" (UID: \"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88\") " Oct 31 01:08:36.211773 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.211742 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-openshift-ca" pod "7b2bcb6f-d7da-4e7e-92b2-9817ab886d88" (UID: "7b2bcb6f-d7da-4e7e-92b2-9817ab886d88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:36.211992 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.211974 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:36.213511 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.213491 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-xks-ca" pod "7b2bcb6f-d7da-4e7e-92b2-9817ab886d88" (UID: "7b2bcb6f-d7da-4e7e-92b2-9817ab886d88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:36.213793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.213772 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-kube-api-access-7bvnm" pod "7b2bcb6f-d7da-4e7e-92b2-9817ab886d88" (UID: "7b2bcb6f-d7da-4e7e-92b2-9817ab886d88"). InnerVolumeSpecName "kube-api-access-7bvnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:36.312545 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.312519 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bvnm\" (UniqueName: \"kubernetes.io/projected/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-kube-api-access-7bvnm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:36.312545 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.312545 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:36.951867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.951798 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" Oct 31 01:08:36.951867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.951803 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5" event={"ID":"7b2bcb6f-d7da-4e7e-92b2-9817ab886d88","Type":"ContainerDied","Data":"d189fc6314011d118aa3d2b2cfc442bca4f1d11886ad43dfde29f6c59ffde560"} Oct 31 01:08:36.951867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:36.951847 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d189fc6314011d118aa3d2b2cfc442bca4f1d11886ad43dfde29f6c59ffde560" Oct 31 01:08:39.493993 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.493963 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx"] Oct 31 01:08:39.494481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.494468 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7b2bcb6f-d7da-4e7e-92b2-9817ab886d88" containerName="revoke-keys" Oct 31 01:08:39.494528 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.494482 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2bcb6f-d7da-4e7e-92b2-9817ab886d88" containerName="revoke-keys" Oct 31 01:08:39.494574 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.494565 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2bcb6f-d7da-4e7e-92b2-9817ab886d88" containerName="revoke-keys" Oct 31 01:08:39.498090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.498075 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.552935 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.552913 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx"] Oct 31 01:08:39.598018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.597990 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:39.598018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.597990 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:39.598211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.598039 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:39.598475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.598456 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:08:39.655754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.655725 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.655883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.655862 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkh4x\" (UniqueName: \"kubernetes.io/projected/11ce8a94-ab67-48f2-9812-d88af1402f4e-kube-api-access-kkh4x\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.656004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.655985 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.756756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.756671 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.756756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.756747 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.756912 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.756834 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkh4x\" (UniqueName: \"kubernetes.io/projected/11ce8a94-ab67-48f2-9812-d88af1402f4e-kube-api-access-kkh4x\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.757382 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.757361 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.759184 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.759166 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:39.816104 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:39.816076 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkh4x\" (UniqueName: \"kubernetes.io/projected/11ce8a94-ab67-48f2-9812-d88af1402f4e-kube-api-access-kkh4x\") pod \"maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:40.111644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.111619 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:40.486348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.486261 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx"] Oct 31 01:08:40.491057 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:40.491031 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ce8a94_ab67_48f2_9812_d88af1402f4e.slice/crio-24bf8044060c52dc4169f602e6bdc0d71a6374eefe5b16b4104ff54c5f87833b WatchSource:0}: Error finding container 24bf8044060c52dc4169f602e6bdc0d71a6374eefe5b16b4104ff54c5f87833b: Status 404 returned error can't find the container with id 24bf8044060c52dc4169f602e6bdc0d71a6374eefe5b16b4104ff54c5f87833b Oct 31 01:08:40.715165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.715129 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq"] Oct 31 01:08:40.723381 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.723362 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.775152 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.775094 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq"] Oct 31 01:08:40.867504 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.867481 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.867625 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.867543 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnq26\" (UniqueName: \"kubernetes.io/projected/164c731b-0862-4031-9a2f-7c08752ca78a-kube-api-access-jnq26\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.867768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.867745 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.969159 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.969136 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.969275 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.969239 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.969320 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.969285 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnq26\" (UniqueName: \"kubernetes.io/projected/164c731b-0862-4031-9a2f-7c08752ca78a-kube-api-access-jnq26\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.969790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.969768 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.971565 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.971546 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:40.988881 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.988854 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" event={"ID":"11ce8a94-ab67-48f2-9812-d88af1402f4e","Type":"ContainerStarted","Data":"2ab3da2ee92f45d891e64b68f6b70686fdff3e7d17eba631526e1d5509c7f3a3"} Oct 31 01:08:40.988974 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:40.988899 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" event={"ID":"11ce8a94-ab67-48f2-9812-d88af1402f4e","Type":"ContainerStarted","Data":"24bf8044060c52dc4169f602e6bdc0d71a6374eefe5b16b4104ff54c5f87833b"} Oct 31 01:08:41.028296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.028246 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnq26\" (UniqueName: \"kubernetes.io/projected/164c731b-0862-4031-9a2f-7c08752ca78a-kube-api-access-jnq26\") pod \"maas-api-revoke-sub-ae3770b5-8xzsq\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:41.036151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.036127 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:41.108038 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.107979 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" podStartSLOduration=2.107960063 podStartE2EDuration="2.107960063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:41.101943667 +0000 UTC m=+2402.638085977" watchObservedRunningTime="2026-10-31 01:08:41.107960063 +0000 UTC m=+2402.644102369" Oct 31 01:08:41.166507 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.166479 5719 scope.go:122] "RemoveContainer" containerID="63e38c6e47a602bb7d9c8f9cb136c06aa700f35998362c6f0b1f4bc5106d9ac5" Oct 31 01:08:41.407662 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.407638 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq"] Oct 31 01:08:41.458683 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:41.458659 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164c731b_0862_4031_9a2f_7c08752ca78a.slice/crio-85d92cfa970eb3378d62f72d54f58d86c3a70bf260e3158d5c8ae9d1d0b5ce46 WatchSource:0}: Error finding container 85d92cfa970eb3378d62f72d54f58d86c3a70bf260e3158d5c8ae9d1d0b5ce46: Status 404 returned error can't find the container with id 85d92cfa970eb3378d62f72d54f58d86c3a70bf260e3158d5c8ae9d1d0b5ce46 Oct 31 01:08:41.997126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.997041 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" event={"ID":"164c731b-0862-4031-9a2f-7c08752ca78a","Type":"ContainerStarted","Data":"274aebdfeb325ac3645cb631bf9f4b87c2379b3b4a7aa29717ee5128bd80993d"} Oct 31 01:08:41.997538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.997135 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" event={"ID":"164c731b-0862-4031-9a2f-7c08752ca78a","Type":"ContainerStarted","Data":"85d92cfa970eb3378d62f72d54f58d86c3a70bf260e3158d5c8ae9d1d0b5ce46"} Oct 31 01:08:41.998761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.998737 5719 generic.go:356] "Generic (PLEG): container finished" podID="11ce8a94-ab67-48f2-9812-d88af1402f4e" containerID="2ab3da2ee92f45d891e64b68f6b70686fdff3e7d17eba631526e1d5509c7f3a3" exitCode=0 Oct 31 01:08:41.998826 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:41.998798 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" event={"ID":"11ce8a94-ab67-48f2-9812-d88af1402f4e","Type":"ContainerDied","Data":"2ab3da2ee92f45d891e64b68f6b70686fdff3e7d17eba631526e1d5509c7f3a3"} Oct 31 01:08:42.109746 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:42.109674 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" podStartSLOduration=2.109661169 podStartE2EDuration="2.109661169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:42.108475552 +0000 UTC m=+2403.644617858" watchObservedRunningTime="2026-10-31 01:08:42.109661169 +0000 UTC m=+2403.645803475" Oct 31 01:08:43.007530 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.007499 5719 generic.go:356] "Generic (PLEG): container finished" podID="164c731b-0862-4031-9a2f-7c08752ca78a" containerID="274aebdfeb325ac3645cb631bf9f4b87c2379b3b4a7aa29717ee5128bd80993d" exitCode=0 Oct 31 01:08:43.008015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.007577 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" event={"ID":"164c731b-0862-4031-9a2f-7c08752ca78a","Type":"ContainerDied","Data":"274aebdfeb325ac3645cb631bf9f4b87c2379b3b4a7aa29717ee5128bd80993d"} Oct 31 01:08:43.161065 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.161038 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:43.295028 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.294954 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ce8a94-ab67-48f2-9812-d88af1402f4e-kube-api-access-kkh4x\" (UniqueName: \"kubernetes.io/projected/11ce8a94-ab67-48f2-9812-d88af1402f4e-kube-api-access-kkh4x\") pod \"11ce8a94-ab67-48f2-9812-d88af1402f4e\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " Oct 31 01:08:43.295028 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.295002 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-openshift-ca\") pod \"11ce8a94-ab67-48f2-9812-d88af1402f4e\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " Oct 31 01:08:43.295210 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.295129 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-xks-ca\") pod \"11ce8a94-ab67-48f2-9812-d88af1402f4e\" (UID: \"11ce8a94-ab67-48f2-9812-d88af1402f4e\") " Oct 31 01:08:43.295463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.295434 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-openshift-ca" pod "11ce8a94-ab67-48f2-9812-d88af1402f4e" (UID: "11ce8a94-ab67-48f2-9812-d88af1402f4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:43.297472 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.297445 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-xks-ca" pod "11ce8a94-ab67-48f2-9812-d88af1402f4e" (UID: "11ce8a94-ab67-48f2-9812-d88af1402f4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:43.301867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.301838 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ce8a94-ab67-48f2-9812-d88af1402f4e-kube-api-access-kkh4x" pod "11ce8a94-ab67-48f2-9812-d88af1402f4e" (UID: "11ce8a94-ab67-48f2-9812-d88af1402f4e"). InnerVolumeSpecName "kube-api-access-kkh4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:43.396926 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.396900 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:43.396926 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.396922 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11ce8a94-ab67-48f2-9812-d88af1402f4e-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:43.397064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:43.396931 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkh4x\" (UniqueName: \"kubernetes.io/projected/11ce8a94-ab67-48f2-9812-d88af1402f4e-kube-api-access-kkh4x\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.015503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.015467 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" event={"ID":"11ce8a94-ab67-48f2-9812-d88af1402f4e","Type":"ContainerDied","Data":"24bf8044060c52dc4169f602e6bdc0d71a6374eefe5b16b4104ff54c5f87833b"} Oct 31 01:08:44.015503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.015484 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx" Oct 31 01:08:44.015983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.015516 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24bf8044060c52dc4169f602e6bdc0d71a6374eefe5b16b4104ff54c5f87833b" Oct 31 01:08:44.169656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.169634 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:44.203791 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.203767 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-xks-ca\") pod \"164c731b-0862-4031-9a2f-7c08752ca78a\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " Oct 31 01:08:44.203913 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.203868 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-openshift-ca\") pod \"164c731b-0862-4031-9a2f-7c08752ca78a\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " Oct 31 01:08:44.203986 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.203959 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164c731b-0862-4031-9a2f-7c08752ca78a-kube-api-access-jnq26\" (UniqueName: \"kubernetes.io/projected/164c731b-0862-4031-9a2f-7c08752ca78a-kube-api-access-jnq26\") pod \"164c731b-0862-4031-9a2f-7c08752ca78a\" (UID: \"164c731b-0862-4031-9a2f-7c08752ca78a\") " Oct 31 01:08:44.204207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.204185 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-openshift-ca" pod "164c731b-0862-4031-9a2f-7c08752ca78a" (UID: "164c731b-0862-4031-9a2f-7c08752ca78a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:44.204344 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.204328 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.205843 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.205822 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-xks-ca" pod "164c731b-0862-4031-9a2f-7c08752ca78a" (UID: "164c731b-0862-4031-9a2f-7c08752ca78a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:44.206132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.206113 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164c731b-0862-4031-9a2f-7c08752ca78a-kube-api-access-jnq26" pod "164c731b-0862-4031-9a2f-7c08752ca78a" (UID: "164c731b-0862-4031-9a2f-7c08752ca78a"). InnerVolumeSpecName "kube-api-access-jnq26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:44.305575 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.305511 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164c731b-0862-4031-9a2f-7c08752ca78a-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:44.305575 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:44.305537 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnq26\" (UniqueName: \"kubernetes.io/projected/164c731b-0862-4031-9a2f-7c08752ca78a-kube-api-access-jnq26\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:45.023103 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:45.023019 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" event={"ID":"164c731b-0862-4031-9a2f-7c08752ca78a","Type":"ContainerDied","Data":"85d92cfa970eb3378d62f72d54f58d86c3a70bf260e3158d5c8ae9d1d0b5ce46"} Oct 31 01:08:45.023103 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:45.023044 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq" Oct 31 01:08:45.023103 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:45.023073 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d92cfa970eb3378d62f72d54f58d86c3a70bf260e3158d5c8ae9d1d0b5ce46" Oct 31 01:08:46.406504 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.406471 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh"] Oct 31 01:08:46.406990 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.406975 5719 cpu_manager.go:414] "containerMap: removing container" podUID="164c731b-0862-4031-9a2f-7c08752ca78a" containerName="revoke-keys" Oct 31 01:08:46.407031 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.406990 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164c731b-0862-4031-9a2f-7c08752ca78a" containerName="revoke-keys" Oct 31 01:08:46.407031 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.407011 5719 cpu_manager.go:414] "containerMap: removing container" podUID="11ce8a94-ab67-48f2-9812-d88af1402f4e" containerName="revoke-keys" Oct 31 01:08:46.407031 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.407016 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ce8a94-ab67-48f2-9812-d88af1402f4e" containerName="revoke-keys" Oct 31 01:08:46.407180 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.407108 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="164c731b-0862-4031-9a2f-7c08752ca78a" containerName="revoke-keys" Oct 31 01:08:46.407180 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.407120 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ce8a94-ab67-48f2-9812-d88af1402f4e" containerName="revoke-keys" Oct 31 01:08:46.410585 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.410569 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.426119 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.426079 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.426343 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.426321 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.426486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.426472 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5bdd\" (UniqueName: \"kubernetes.io/projected/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-kube-api-access-d5bdd\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.510110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.510084 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:46.510110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.510085 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:46.510316 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.510115 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:46.510316 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.510140 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:08:46.513365 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.513339 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh"] Oct 31 01:08:46.528063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.528038 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5bdd\" (UniqueName: \"kubernetes.io/projected/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-kube-api-access-d5bdd\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.528247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.528228 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.528286 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.528264 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.528844 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.528825 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.530698 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.530675 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.586623 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.586598 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5bdd\" (UniqueName: \"kubernetes.io/projected/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-kube-api-access-d5bdd\") pod \"maas-api-revoke-sub-5e0c7894-86rzh\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:46.725259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:46.725201 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:47.101011 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:47.100981 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh"] Oct 31 01:08:47.105583 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:47.105559 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod468a6d5c_bb0b_4442_9ac9_9a90ef20eb67.slice/crio-72f10930974e8d3e7e8c263b62fe6fef8402fbb383c933595088eca4f4aff0d9 WatchSource:0}: Error finding container 72f10930974e8d3e7e8c263b62fe6fef8402fbb383c933595088eca4f4aff0d9: Status 404 returned error can't find the container with id 72f10930974e8d3e7e8c263b62fe6fef8402fbb383c933595088eca4f4aff0d9 Oct 31 01:08:48.043678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:48.043653 5719 generic.go:356] "Generic (PLEG): container finished" podID="468a6d5c-bb0b-4442-9ac9-9a90ef20eb67" containerID="14e306aea031070ceac7aa5423df5c6c22332127865ea00b483f2faa470911a8" exitCode=0 Oct 31 01:08:48.044013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:48.043716 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" event={"ID":"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67","Type":"ContainerDied","Data":"14e306aea031070ceac7aa5423df5c6c22332127865ea00b483f2faa470911a8"} Oct 31 01:08:48.044013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:48.043758 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" event={"ID":"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67","Type":"ContainerStarted","Data":"72f10930974e8d3e7e8c263b62fe6fef8402fbb383c933595088eca4f4aff0d9"} Oct 31 01:08:49.197504 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.197482 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:49.255947 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.255918 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-openshift-ca\") pod \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " Oct 31 01:08:49.256064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.256005 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-kube-api-access-d5bdd\" (UniqueName: \"kubernetes.io/projected/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-kube-api-access-d5bdd\") pod \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " Oct 31 01:08:49.256153 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.256137 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-xks-ca\") pod \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\" (UID: \"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67\") " Oct 31 01:08:49.256290 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.256260 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-openshift-ca" pod "468a6d5c-bb0b-4442-9ac9-9a90ef20eb67" (UID: "468a6d5c-bb0b-4442-9ac9-9a90ef20eb67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:49.256555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.256537 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:49.258062 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.258045 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-kube-api-access-d5bdd" pod "468a6d5c-bb0b-4442-9ac9-9a90ef20eb67" (UID: "468a6d5c-bb0b-4442-9ac9-9a90ef20eb67"). InnerVolumeSpecName "kube-api-access-d5bdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:49.258180 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.258163 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-xks-ca" pod "468a6d5c-bb0b-4442-9ac9-9a90ef20eb67" (UID: "468a6d5c-bb0b-4442-9ac9-9a90ef20eb67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:08:49.357739 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.357686 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:49.357739 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:49.357737 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5bdd\" (UniqueName: \"kubernetes.io/projected/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67-kube-api-access-d5bdd\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:50.057299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:50.057263 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" event={"ID":"468a6d5c-bb0b-4442-9ac9-9a90ef20eb67","Type":"ContainerDied","Data":"72f10930974e8d3e7e8c263b62fe6fef8402fbb383c933595088eca4f4aff0d9"} Oct 31 01:08:50.057463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:50.057340 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72f10930974e8d3e7e8c263b62fe6fef8402fbb383c933595088eca4f4aff0d9" Oct 31 01:08:50.057463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:50.057274 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh" Oct 31 01:08:51.002762 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:51.002734 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh"] Oct 31 01:08:51.104309 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:51.104281 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl"] Oct 31 01:08:56.003315 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.003253 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" podUID="f81e8be5-748b-4a9f-98c3-909da65cf65a" containerName="payload-processing" containerID="cri-o://e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09" gracePeriod=25 Oct 31 01:08:56.105037 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.104989 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" podUID="1d1d3962-918c-480b-8ab0-35d114504e31" containerName="payload-pre-processing" containerID="cri-o://2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639" gracePeriod=25 Oct 31 01:08:56.259951 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.259895 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:08:56.334570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.334542 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81e8be5-748b-4a9f-98c3-909da65cf65a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81e8be5-748b-4a9f-98c3-909da65cf65a-plugins-config-volume\") pod \"f81e8be5-748b-4a9f-98c3-909da65cf65a\" (UID: \"f81e8be5-748b-4a9f-98c3-909da65cf65a\") " Oct 31 01:08:56.334748 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.334685 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81e8be5-748b-4a9f-98c3-909da65cf65a-kube-api-access-9rvfj\" (UniqueName: \"kubernetes.io/projected/f81e8be5-748b-4a9f-98c3-909da65cf65a-kube-api-access-9rvfj\") pod \"f81e8be5-748b-4a9f-98c3-909da65cf65a\" (UID: \"f81e8be5-748b-4a9f-98c3-909da65cf65a\") " Oct 31 01:08:56.334931 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.334909 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81e8be5-748b-4a9f-98c3-909da65cf65a-plugins-config-volume" pod "f81e8be5-748b-4a9f-98c3-909da65cf65a" (UID: "f81e8be5-748b-4a9f-98c3-909da65cf65a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:56.335089 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.335072 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81e8be5-748b-4a9f-98c3-909da65cf65a-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:56.337302 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.337275 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81e8be5-748b-4a9f-98c3-909da65cf65a-kube-api-access-9rvfj" pod "f81e8be5-748b-4a9f-98c3-909da65cf65a" (UID: "f81e8be5-748b-4a9f-98c3-909da65cf65a"). InnerVolumeSpecName "kube-api-access-9rvfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:56.360016 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.359997 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:08:56.435906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.435883 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1d3962-918c-480b-8ab0-35d114504e31-kube-api-access-gjdhh\" (UniqueName: \"kubernetes.io/projected/1d1d3962-918c-480b-8ab0-35d114504e31-kube-api-access-gjdhh\") pod \"1d1d3962-918c-480b-8ab0-35d114504e31\" (UID: \"1d1d3962-918c-480b-8ab0-35d114504e31\") " Oct 31 01:08:56.436042 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.435945 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1d3962-918c-480b-8ab0-35d114504e31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1d3962-918c-480b-8ab0-35d114504e31-plugins-config-volume\") pod \"1d1d3962-918c-480b-8ab0-35d114504e31\" (UID: \"1d1d3962-918c-480b-8ab0-35d114504e31\") " Oct 31 01:08:56.436321 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.436294 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1d3962-918c-480b-8ab0-35d114504e31-plugins-config-volume" pod "1d1d3962-918c-480b-8ab0-35d114504e31" (UID: "1d1d3962-918c-480b-8ab0-35d114504e31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:08:56.436430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.436343 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rvfj\" (UniqueName: \"kubernetes.io/projected/f81e8be5-748b-4a9f-98c3-909da65cf65a-kube-api-access-9rvfj\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:56.438024 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.438006 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1d3962-918c-480b-8ab0-35d114504e31-kube-api-access-gjdhh" pod "1d1d3962-918c-480b-8ab0-35d114504e31" (UID: "1d1d3962-918c-480b-8ab0-35d114504e31"). InnerVolumeSpecName "kube-api-access-gjdhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:08:56.537797 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.537732 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdhh\" (UniqueName: \"kubernetes.io/projected/1d1d3962-918c-480b-8ab0-35d114504e31-kube-api-access-gjdhh\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:56.537797 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:56.537756 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1d3962-918c-480b-8ab0-35d114504e31-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:08:57.109678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.109642 5719 generic.go:356] "Generic (PLEG): container finished" podID="1d1d3962-918c-480b-8ab0-35d114504e31" containerID="2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639" exitCode=0 Oct 31 01:08:57.110189 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.109727 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" Oct 31 01:08:57.110189 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.109741 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" event={"ID":"1d1d3962-918c-480b-8ab0-35d114504e31","Type":"ContainerDied","Data":"2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639"} Oct 31 01:08:57.110189 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.109807 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl" event={"ID":"1d1d3962-918c-480b-8ab0-35d114504e31","Type":"ContainerDied","Data":"8a1ad46efa74df76b86da35f29916a536e41e4c19f7fcb4bfd3987fe7f793ac7"} Oct 31 01:08:57.110189 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.109840 5719 scope.go:122] "RemoveContainer" containerID="2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639" Oct 31 01:08:57.111877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.111757 5719 generic.go:356] "Generic (PLEG): container finished" podID="f81e8be5-748b-4a9f-98c3-909da65cf65a" containerID="e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09" exitCode=0 Oct 31 01:08:57.111877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.111812 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" Oct 31 01:08:57.112009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.111960 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" event={"ID":"f81e8be5-748b-4a9f-98c3-909da65cf65a","Type":"ContainerDied","Data":"e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09"} Oct 31 01:08:57.112009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.112004 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh" event={"ID":"f81e8be5-748b-4a9f-98c3-909da65cf65a","Type":"ContainerDied","Data":"d3fd29138c7c5576b32661ae870626dbabe15e902bce52ba74b70b628a26c35f"} Oct 31 01:08:57.123836 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.123815 5719 scope.go:122] "RemoveContainer" containerID="2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639" Oct 31 01:08:57.124362 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:08:57.124335 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639\": container with ID starting with 2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639 not found: ID does not exist" containerID="2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639" Oct 31 01:08:57.124577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.124521 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639"} err="failed to get container status \"2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639\": rpc error: code = NotFound desc = could not find container \"2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639\": container with ID starting with 2ab50207cf89c10e003a3aeac23cc5a73701bda984808bcf6e13c907e0165639 not found: ID does not exist" Oct 31 01:08:57.124805 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.124678 5719 scope.go:122] "RemoveContainer" containerID="e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09" Oct 31 01:08:57.143743 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.142844 5719 scope.go:122] "RemoveContainer" containerID="e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09" Oct 31 01:08:57.143743 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:08:57.143342 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09\": container with ID starting with e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09 not found: ID does not exist" containerID="e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09" Oct 31 01:08:57.143743 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.143393 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09"} err="failed to get container status \"e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09\": rpc error: code = NotFound desc = could not find container \"e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09\": container with ID starting with e117bac4d1e251cb21aa3136cd813b4ac2f76c910442def9c1322b1de3ca6d09 not found: ID does not exist" Oct 31 01:08:57.149328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.149298 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7"] Oct 31 01:08:57.150092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150075 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f81e8be5-748b-4a9f-98c3-909da65cf65a" containerName="payload-processing" Oct 31 01:08:57.150226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150094 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81e8be5-748b-4a9f-98c3-909da65cf65a" containerName="payload-processing" Oct 31 01:08:57.150226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150125 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1d1d3962-918c-480b-8ab0-35d114504e31" containerName="payload-pre-processing" Oct 31 01:08:57.150226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150133 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1d3962-918c-480b-8ab0-35d114504e31" containerName="payload-pre-processing" Oct 31 01:08:57.150226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150174 5719 cpu_manager.go:414] "containerMap: removing container" podUID="468a6d5c-bb0b-4442-9ac9-9a90ef20eb67" containerName="revoke-keys" Oct 31 01:08:57.150226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150181 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468a6d5c-bb0b-4442-9ac9-9a90ef20eb67" containerName="revoke-keys" Oct 31 01:08:57.150520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150322 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81e8be5-748b-4a9f-98c3-909da65cf65a" containerName="payload-processing" Oct 31 01:08:57.150520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150339 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1d3962-918c-480b-8ab0-35d114504e31" containerName="payload-pre-processing" Oct 31 01:08:57.150520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.150354 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="468a6d5c-bb0b-4442-9ac9-9a90ef20eb67" containerName="revoke-keys" Oct 31 01:08:57.155713 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.155679 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.204417 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.204369 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7"] Oct 31 01:08:57.208261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.208230 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:08:57.208475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.208263 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:08:57.208576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.208263 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:08:57.208812 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.208789 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:08:57.245098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.245064 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.245273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.245164 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.245360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.245342 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fmw8\" (UniqueName: \"kubernetes.io/projected/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-kube-api-access-5fmw8\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.309973 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.309880 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh"] Oct 31 01:08:57.314806 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.314733 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3f25cd-59987c57ff-949wh"] Oct 31 01:08:57.347195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.347156 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.347306 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.347221 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.347396 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.347317 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fmw8\" (UniqueName: \"kubernetes.io/projected/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-kube-api-access-5fmw8\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.348337 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.348312 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.350790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.350736 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.412588 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.412555 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fmw8\" (UniqueName: \"kubernetes.io/projected/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-kube-api-access-5fmw8\") pod \"maas-api-revoke-sub-ceb285d7-mvxk7\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.418385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.418356 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl"] Oct 31 01:08:57.423260 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.423232 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3f25cd-65b4c85fff-xbpcl"] Oct 31 01:08:57.470897 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.470754 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:08:57.751718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.751159 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5"] Oct 31 01:08:57.796404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.796375 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5"] Oct 31 01:08:57.796601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.796495 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:57.919716 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.909301 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7"] Oct 31 01:08:57.951771 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.943755 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdn62\" (UniqueName: \"kubernetes.io/projected/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-kube-api-access-zdn62\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:57.951771 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.943861 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:57.951771 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:57.943988 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:58.054443 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.054405 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:58.055944 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.055919 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:58.056538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.056515 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdn62\" (UniqueName: \"kubernetes.io/projected/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-kube-api-access-zdn62\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:58.057089 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.055720 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:58.076159 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.076100 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:58.155587 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.155554 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdn62\" (UniqueName: \"kubernetes.io/projected/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-kube-api-access-zdn62\") pod \"maas-api-revoke-sub-b5789ca9-fm9d5\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:58.216313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.216276 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" event={"ID":"abcaccce-b4bf-4db4-bae2-8c3b585ebeac","Type":"ContainerStarted","Data":"ce20bbc28acb0edd987eab0a4011b15e8574e0492ef5a1f5836db93891d49835"} Oct 31 01:08:58.445766 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.445355 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:08:58.668718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.665149 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1d3962-918c-480b-8ab0-35d114504e31" path="/var/lib/kubelet/pods/1d1d3962-918c-480b-8ab0-35d114504e31/volumes" Oct 31 01:08:58.668718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:58.665686 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81e8be5-748b-4a9f-98c3-909da65cf65a" path="/var/lib/kubelet/pods/f81e8be5-748b-4a9f-98c3-909da65cf65a/volumes" Oct 31 01:08:59.036949 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:59.036878 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5"] Oct 31 01:08:59.041357 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:08:59.041325 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e932d2c_21ce_48d2_87c1_98c9b64d64a0.slice/crio-3a621c25c7a3e213d78bfa0ca205cdb6fd92563ff97ca6d498c66152e16a6d8b WatchSource:0}: Error finding container 3a621c25c7a3e213d78bfa0ca205cdb6fd92563ff97ca6d498c66152e16a6d8b: Status 404 returned error can't find the container with id 3a621c25c7a3e213d78bfa0ca205cdb6fd92563ff97ca6d498c66152e16a6d8b Oct 31 01:08:59.224778 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:59.224739 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" event={"ID":"8e932d2c-21ce-48d2-87c1-98c9b64d64a0","Type":"ContainerStarted","Data":"3a621c25c7a3e213d78bfa0ca205cdb6fd92563ff97ca6d498c66152e16a6d8b"} Oct 31 01:08:59.226465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:59.226435 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" event={"ID":"abcaccce-b4bf-4db4-bae2-8c3b585ebeac","Type":"ContainerStarted","Data":"a67b30fed721697a5e1f113a52a64a3642c2d2bb7efa38c18b64932a4331f2a5"} Oct 31 01:08:59.341449 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:08:59.341404 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" podStartSLOduration=2.341389501 podStartE2EDuration="2.341389501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:59.340059252 +0000 UTC m=+2420.876201557" watchObservedRunningTime="2026-10-31 01:08:59.341389501 +0000 UTC m=+2420.877531805" Oct 31 01:09:00.234809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:00.234721 5719 generic.go:356] "Generic (PLEG): container finished" podID="8e932d2c-21ce-48d2-87c1-98c9b64d64a0" containerID="b684b39ed35ea81254778d995c221e4b9736cd9742ff9c9494503d8757abe281" exitCode=0 Oct 31 01:09:00.235252 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:00.234807 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" event={"ID":"8e932d2c-21ce-48d2-87c1-98c9b64d64a0","Type":"ContainerDied","Data":"b684b39ed35ea81254778d995c221e4b9736cd9742ff9c9494503d8757abe281"} Oct 31 01:09:00.236586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:00.236556 5719 generic.go:356] "Generic (PLEG): container finished" podID="abcaccce-b4bf-4db4-bae2-8c3b585ebeac" containerID="a67b30fed721697a5e1f113a52a64a3642c2d2bb7efa38c18b64932a4331f2a5" exitCode=0 Oct 31 01:09:00.236702 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:00.236623 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" event={"ID":"abcaccce-b4bf-4db4-bae2-8c3b585ebeac","Type":"ContainerDied","Data":"a67b30fed721697a5e1f113a52a64a3642c2d2bb7efa38c18b64932a4331f2a5"} Oct 31 01:09:01.403658 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.403634 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:09:01.406918 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.406896 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:09:01.537524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.537450 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-xks-ca\") pod \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " Oct 31 01:09:01.537524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.537508 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-kube-api-access-zdn62\" (UniqueName: \"kubernetes.io/projected/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-kube-api-access-zdn62\") pod \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " Oct 31 01:09:01.537744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.537560 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-kube-api-access-5fmw8\" (UniqueName: \"kubernetes.io/projected/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-kube-api-access-5fmw8\") pod \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " Oct 31 01:09:01.537744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.537594 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-openshift-ca\") pod \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\" (UID: \"abcaccce-b4bf-4db4-bae2-8c3b585ebeac\") " Oct 31 01:09:01.537744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.537637 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-xks-ca\") pod \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " Oct 31 01:09:01.537744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.537686 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-openshift-ca\") pod \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\" (UID: \"8e932d2c-21ce-48d2-87c1-98c9b64d64a0\") " Oct 31 01:09:01.538032 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.538007 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-openshift-ca" pod "abcaccce-b4bf-4db4-bae2-8c3b585ebeac" (UID: "abcaccce-b4bf-4db4-bae2-8c3b585ebeac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:01.538391 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.538323 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.538391 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.538323 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-openshift-ca" pod "8e932d2c-21ce-48d2-87c1-98c9b64d64a0" (UID: "8e932d2c-21ce-48d2-87c1-98c9b64d64a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:01.539889 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.539863 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-kube-api-access-5fmw8" pod "abcaccce-b4bf-4db4-bae2-8c3b585ebeac" (UID: "abcaccce-b4bf-4db4-bae2-8c3b585ebeac"). InnerVolumeSpecName "kube-api-access-5fmw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:01.541826 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.541791 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-xks-ca" pod "abcaccce-b4bf-4db4-bae2-8c3b585ebeac" (UID: "abcaccce-b4bf-4db4-bae2-8c3b585ebeac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:01.541912 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.541855 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-xks-ca" pod "8e932d2c-21ce-48d2-87c1-98c9b64d64a0" (UID: "8e932d2c-21ce-48d2-87c1-98c9b64d64a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:01.541912 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.541895 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-kube-api-access-zdn62" pod "8e932d2c-21ce-48d2-87c1-98c9b64d64a0" (UID: "8e932d2c-21ce-48d2-87c1-98c9b64d64a0"). InnerVolumeSpecName "kube-api-access-zdn62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:01.639429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.639404 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fmw8\" (UniqueName: \"kubernetes.io/projected/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-kube-api-access-5fmw8\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.639429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.639427 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.639592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.639437 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.639592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.639446 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcaccce-b4bf-4db4-bae2-8c3b585ebeac-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:01.639592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:01.639456 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdn62\" (UniqueName: \"kubernetes.io/projected/8e932d2c-21ce-48d2-87c1-98c9b64d64a0-kube-api-access-zdn62\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:02.250755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:02.250712 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" event={"ID":"8e932d2c-21ce-48d2-87c1-98c9b64d64a0","Type":"ContainerDied","Data":"3a621c25c7a3e213d78bfa0ca205cdb6fd92563ff97ca6d498c66152e16a6d8b"} Oct 31 01:09:02.251013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:02.250772 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a621c25c7a3e213d78bfa0ca205cdb6fd92563ff97ca6d498c66152e16a6d8b" Oct 31 01:09:02.251013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:02.250729 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5" Oct 31 01:09:02.252272 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:02.252252 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" Oct 31 01:09:02.252272 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:02.252257 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7" event={"ID":"abcaccce-b4bf-4db4-bae2-8c3b585ebeac","Type":"ContainerDied","Data":"ce20bbc28acb0edd987eab0a4011b15e8574e0492ef5a1f5836db93891d49835"} Oct 31 01:09:02.252418 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:02.252292 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce20bbc28acb0edd987eab0a4011b15e8574e0492ef5a1f5836db93891d49835" Oct 31 01:09:03.010058 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:09:03.010021 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:09:13.743877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:13.743841 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:09:14.374876 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.374830 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk"] Oct 31 01:09:14.375782 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.375757 5719 cpu_manager.go:414] "containerMap: removing container" podUID="abcaccce-b4bf-4db4-bae2-8c3b585ebeac" containerName="revoke-keys" Oct 31 01:09:14.375942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.375929 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcaccce-b4bf-4db4-bae2-8c3b585ebeac" containerName="revoke-keys" Oct 31 01:09:14.376046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.376037 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8e932d2c-21ce-48d2-87c1-98c9b64d64a0" containerName="revoke-keys" Oct 31 01:09:14.376130 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.376118 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e932d2c-21ce-48d2-87c1-98c9b64d64a0" containerName="revoke-keys" Oct 31 01:09:14.376384 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.376361 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcaccce-b4bf-4db4-bae2-8c3b585ebeac" containerName="revoke-keys" Oct 31 01:09:14.376384 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.376386 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e932d2c-21ce-48d2-87c1-98c9b64d64a0" containerName="revoke-keys" Oct 31 01:09:14.380975 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.380938 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.440201 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.440176 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk"] Oct 31 01:09:14.482113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.482079 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:09:14.482290 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.482271 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:09:14.482365 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.482350 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:09:14.484510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.484490 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:09:14.485968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.485944 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.486094 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.486073 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ggp\" (UniqueName: \"kubernetes.io/projected/9f2c6fd9-2445-4f03-a860-c279276fc1cc-kube-api-access-j2ggp\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.486257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.486240 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.586965 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.586925 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.587191 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.587007 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.587191 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.587089 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ggp\" (UniqueName: \"kubernetes.io/projected/9f2c6fd9-2445-4f03-a860-c279276fc1cc-kube-api-access-j2ggp\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.588214 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.588191 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.603093 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.597836 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.649667 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.649594 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ggp\" (UniqueName: \"kubernetes.io/projected/9f2c6fd9-2445-4f03-a860-c279276fc1cc-kube-api-access-j2ggp\") pod \"maas-api-revoke-sub-c83aa7b5-rlqlk\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:14.703719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:14.702854 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:15.199925 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:15.199845 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f"] Oct 31 01:09:15.200395 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:15.200090 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" containerID="cri-o://1f37b8b9a3d613e055cf0caa5767e95d1fa1850509cb90c3b4836ef1e3f02df7" gracePeriod=30 Oct 31 01:09:15.470060 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:15.470007 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk"] Oct 31 01:09:16.301161 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.301133 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq"] Oct 31 01:09:16.305019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.305003 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.361804 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.361773 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq"] Oct 31 01:09:16.367959 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.367926 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7p7\" (UniqueName: \"kubernetes.io/projected/cf50bc8c-b168-440f-b4fb-e54075233151-kube-api-access-vc7p7\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.368126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.367971 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.368126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.368093 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.395350 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.395311 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" event={"ID":"9f2c6fd9-2445-4f03-a860-c279276fc1cc","Type":"ContainerStarted","Data":"80e476d80243c60964a655a8cbc615883341f6b3ebc17d5d327fa4ec60bfb7a8"} Oct 31 01:09:16.395502 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.395363 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" event={"ID":"9f2c6fd9-2445-4f03-a860-c279276fc1cc","Type":"ContainerStarted","Data":"749193cb2199da76346bfe3912f1170bea7e49ec125987a7ce6b7f5b0ed4acac"} Oct 31 01:09:16.468969 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.468932 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.469206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.469176 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7p7\" (UniqueName: \"kubernetes.io/projected/cf50bc8c-b168-440f-b4fb-e54075233151-kube-api-access-vc7p7\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.469320 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.469229 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.470041 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.470012 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.471506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.471481 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.528116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.528086 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7p7\" (UniqueName: \"kubernetes.io/projected/cf50bc8c-b168-440f-b4fb-e54075233151-kube-api-access-vc7p7\") pod \"maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.530860 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.530820 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" podStartSLOduration=2.530808264 podStartE2EDuration="2.530808264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:16.528570283 +0000 UTC m=+2438.064712590" watchObservedRunningTime="2026-10-31 01:09:16.530808264 +0000 UTC m=+2438.066950569" Oct 31 01:09:16.618113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.618088 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:16.993848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:16.993768 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq"] Oct 31 01:09:16.997874 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:09:16.997850 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf50bc8c_b168_440f_b4fb_e54075233151.slice/crio-3d0066da0645379eb38c4d9a0983258235d2864ccbaa85633e58fcdffa13191e WatchSource:0}: Error finding container 3d0066da0645379eb38c4d9a0983258235d2864ccbaa85633e58fcdffa13191e: Status 404 returned error can't find the container with id 3d0066da0645379eb38c4d9a0983258235d2864ccbaa85633e58fcdffa13191e Oct 31 01:09:17.052149 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:17.052121 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8"] Oct 31 01:09:17.055313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:17.055291 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0d6b5ab-lr4t8"] Oct 31 01:09:17.403519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:17.403487 5719 generic.go:356] "Generic (PLEG): container finished" podID="9f2c6fd9-2445-4f03-a860-c279276fc1cc" containerID="80e476d80243c60964a655a8cbc615883341f6b3ebc17d5d327fa4ec60bfb7a8" exitCode=0 Oct 31 01:09:17.403978 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:17.403571 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" event={"ID":"9f2c6fd9-2445-4f03-a860-c279276fc1cc","Type":"ContainerDied","Data":"80e476d80243c60964a655a8cbc615883341f6b3ebc17d5d327fa4ec60bfb7a8"} Oct 31 01:09:17.405481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:17.405454 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" event={"ID":"cf50bc8c-b168-440f-b4fb-e54075233151","Type":"ContainerStarted","Data":"b66707efd269bd491d53abbb169f7a2ce1ae61f5a3d3f066fac682125f827f28"} Oct 31 01:09:17.405630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:17.405521 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" event={"ID":"cf50bc8c-b168-440f-b4fb-e54075233151","Type":"ContainerStarted","Data":"3d0066da0645379eb38c4d9a0983258235d2864ccbaa85633e58fcdffa13191e"} Oct 31 01:09:17.626868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:17.626816 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" podStartSLOduration=1.626803122 podStartE2EDuration="1.626803122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:17.625586436 +0000 UTC m=+2439.161728743" watchObservedRunningTime="2026-10-31 01:09:17.626803122 +0000 UTC m=+2439.162945434" Oct 31 01:09:18.413594 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.413555 5719 generic.go:356] "Generic (PLEG): container finished" podID="cf50bc8c-b168-440f-b4fb-e54075233151" containerID="b66707efd269bd491d53abbb169f7a2ce1ae61f5a3d3f066fac682125f827f28" exitCode=0 Oct 31 01:09:18.414003 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.413637 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" event={"ID":"cf50bc8c-b168-440f-b4fb-e54075233151","Type":"ContainerDied","Data":"b66707efd269bd491d53abbb169f7a2ce1ae61f5a3d3f066fac682125f827f28"} Oct 31 01:09:18.568181 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.568157 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:18.594002 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.593972 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2c6fd9-2445-4f03-a860-c279276fc1cc-kube-api-access-j2ggp\" (UniqueName: \"kubernetes.io/projected/9f2c6fd9-2445-4f03-a860-c279276fc1cc-kube-api-access-j2ggp\") pod \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " Oct 31 01:09:18.594162 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.594146 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-xks-ca\") pod \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " Oct 31 01:09:18.597735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.594265 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-openshift-ca\") pod \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\" (UID: \"9f2c6fd9-2445-4f03-a860-c279276fc1cc\") " Oct 31 01:09:18.597735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.594642 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-openshift-ca" pod "9f2c6fd9-2445-4f03-a860-c279276fc1cc" (UID: "9f2c6fd9-2445-4f03-a860-c279276fc1cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:18.597735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.596559 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2c6fd9-2445-4f03-a860-c279276fc1cc-kube-api-access-j2ggp" pod "9f2c6fd9-2445-4f03-a860-c279276fc1cc" (UID: "9f2c6fd9-2445-4f03-a860-c279276fc1cc"). InnerVolumeSpecName "kube-api-access-j2ggp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:18.597735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.596662 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-xks-ca" pod "9f2c6fd9-2445-4f03-a860-c279276fc1cc" (UID: "9f2c6fd9-2445-4f03-a860-c279276fc1cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:18.603622 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.603598 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e133a8-2650-4e40-a7db-a2fa71e7a511" path="/var/lib/kubelet/pods/03e133a8-2650-4e40-a7db-a2fa71e7a511/volumes" Oct 31 01:09:18.695335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.695272 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.695335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.695303 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f2c6fd9-2445-4f03-a860-c279276fc1cc-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:18.695335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:18.695321 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2ggp\" (UniqueName: \"kubernetes.io/projected/9f2c6fd9-2445-4f03-a860-c279276fc1cc-kube-api-access-j2ggp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.422418 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.422392 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" Oct 31 01:09:19.422887 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.422393 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk" event={"ID":"9f2c6fd9-2445-4f03-a860-c279276fc1cc","Type":"ContainerDied","Data":"749193cb2199da76346bfe3912f1170bea7e49ec125987a7ce6b7f5b0ed4acac"} Oct 31 01:09:19.422887 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.422526 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="749193cb2199da76346bfe3912f1170bea7e49ec125987a7ce6b7f5b0ed4acac" Oct 31 01:09:19.568751 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.568728 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:19.606433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.606409 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf50bc8c-b168-440f-b4fb-e54075233151-kube-api-access-vc7p7\" (UniqueName: \"kubernetes.io/projected/cf50bc8c-b168-440f-b4fb-e54075233151-kube-api-access-vc7p7\") pod \"cf50bc8c-b168-440f-b4fb-e54075233151\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " Oct 31 01:09:19.606565 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.606452 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-openshift-ca\") pod \"cf50bc8c-b168-440f-b4fb-e54075233151\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " Oct 31 01:09:19.606645 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.606567 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-xks-ca\") pod \"cf50bc8c-b168-440f-b4fb-e54075233151\" (UID: \"cf50bc8c-b168-440f-b4fb-e54075233151\") " Oct 31 01:09:19.606874 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.606850 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-openshift-ca" pod "cf50bc8c-b168-440f-b4fb-e54075233151" (UID: "cf50bc8c-b168-440f-b4fb-e54075233151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:19.607009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.606989 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.608783 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.608763 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-xks-ca" pod "cf50bc8c-b168-440f-b4fb-e54075233151" (UID: "cf50bc8c-b168-440f-b4fb-e54075233151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:09:19.608930 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.608913 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf50bc8c-b168-440f-b4fb-e54075233151-kube-api-access-vc7p7" pod "cf50bc8c-b168-440f-b4fb-e54075233151" (UID: "cf50bc8c-b168-440f-b4fb-e54075233151"). InnerVolumeSpecName "kube-api-access-vc7p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:19.708527 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.708466 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7p7\" (UniqueName: \"kubernetes.io/projected/cf50bc8c-b168-440f-b4fb-e54075233151-kube-api-access-vc7p7\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:19.708527 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:19.708487 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf50bc8c-b168-440f-b4fb-e54075233151-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.439680 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.438220 5719 generic.go:356] "Generic (PLEG): container finished" podID="5608df70-fa36-4805-8893-1277b632649a" containerID="1f37b8b9a3d613e055cf0caa5767e95d1fa1850509cb90c3b4836ef1e3f02df7" exitCode=0 Oct 31 01:09:20.439680 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.438365 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" event={"ID":"5608df70-fa36-4805-8893-1277b632649a","Type":"ContainerDied","Data":"1f37b8b9a3d613e055cf0caa5767e95d1fa1850509cb90c3b4836ef1e3f02df7"} Oct 31 01:09:20.442308 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.442284 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" Oct 31 01:09:20.442431 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.442314 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq" event={"ID":"cf50bc8c-b168-440f-b4fb-e54075233151","Type":"ContainerDied","Data":"3d0066da0645379eb38c4d9a0983258235d2864ccbaa85633e58fcdffa13191e"} Oct 31 01:09:20.442431 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.442357 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d0066da0645379eb38c4d9a0983258235d2864ccbaa85633e58fcdffa13191e" Oct 31 01:09:20.528207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.528183 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 01:09:20.619170 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619149 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5608df70-fa36-4805-8893-1277b632649a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5608df70-fa36-4805-8893-1277b632649a-istio-podinfo\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619195 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-socket\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619242 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-kube-api-access-thvb9\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-kube-api-access-thvb9\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619270 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-envoy\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619413 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-certs\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619458 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-data\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619591 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619500 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-istio-token\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-istio-token\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619591 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619562 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5608df70-fa36-4805-8893-1277b632649a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5608df70-fa36-4805-8893-1277b632649a-istiod-ca-cert\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619591 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619563 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-socket" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:20.619780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619621 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-credential-socket\") pod \"5608df70-fa36-4805-8893-1277b632649a\" (UID: \"5608df70-fa36-4805-8893-1277b632649a\") " Oct 31 01:09:20.619780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619675 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-certs" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:20.619888 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619782 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-data" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:20.619988 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.619968 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5608df70-fa36-4805-8893-1277b632649a-istiod-ca-cert" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:20.620055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.620003 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-credential-socket" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:20.620186 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.620163 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.620248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.620191 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.620248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.620205 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5608df70-fa36-4805-8893-1277b632649a-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.620248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.620219 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.620248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.620232 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.621854 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.621833 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-istio-token" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:20.621985 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.621970 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-envoy" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:09:20.622046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.621995 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-kube-api-access-thvb9" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "kube-api-access-thvb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:20.622085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.622054 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5608df70-fa36-4805-8893-1277b632649a-istio-podinfo" pod "5608df70-fa36-4805-8893-1277b632649a" (UID: "5608df70-fa36-4805-8893-1277b632649a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:09:20.721261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.721233 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.721261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.721259 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5608df70-fa36-4805-8893-1277b632649a-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.721261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.721269 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thvb9\" (UniqueName: \"kubernetes.io/projected/5608df70-fa36-4805-8893-1277b632649a-kube-api-access-thvb9\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:20.721489 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:20.721278 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5608df70-fa36-4805-8893-1277b632649a-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:21.450353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:21.450324 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" Oct 31 01:09:21.450809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:21.450361 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f" event={"ID":"5608df70-fa36-4805-8893-1277b632649a","Type":"ContainerDied","Data":"4c3380c4b06687cfec8619e3039b0c62a05f275d66957026c78feef73d394865"} Oct 31 01:09:21.450809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:21.450426 5719 scope.go:122] "RemoveContainer" containerID="1f37b8b9a3d613e055cf0caa5767e95d1fa1850509cb90c3b4836ef1e3f02df7" Oct 31 01:09:21.609576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:21.609547 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f"] Oct 31 01:09:21.657320 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:21.657291 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-29ece5-openshift-default-6fcb86dbc-6j72f"] Oct 31 01:09:22.602901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:22.602865 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5608df70-fa36-4805-8893-1277b632649a" path="/var/lib/kubelet/pods/5608df70-fa36-4805-8893-1277b632649a/volumes" Oct 31 01:09:27.813390 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:27.813349 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf"] Oct 31 01:09:29.014797 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:29.014764 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx"] Oct 31 01:09:29.019558 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:29.019525 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3f25cd-dkvpx"] Oct 31 01:09:30.601611 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:30.601575 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ce8a94-ab67-48f2-9812-d88af1402f4e" path="/var/lib/kubelet/pods/11ce8a94-ab67-48f2-9812-d88af1402f4e/volumes" Oct 31 01:09:32.814920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:32.814883 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" podUID="5a34ee25-d63f-4ed2-8a45-304105ec08ee" containerName="payload-processing" containerID="cri-o://bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10" gracePeriod=25 Oct 31 01:09:33.020283 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:09:33.020246 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:09:33.069169 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.069139 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:09:33.143898 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.143864 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a34ee25-d63f-4ed2-8a45-304105ec08ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a34ee25-d63f-4ed2-8a45-304105ec08ee-plugins-config-volume\") pod \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\" (UID: \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\") " Oct 31 01:09:33.144075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.144020 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a34ee25-d63f-4ed2-8a45-304105ec08ee-kube-api-access-fh6fq\" (UniqueName: \"kubernetes.io/projected/5a34ee25-d63f-4ed2-8a45-304105ec08ee-kube-api-access-fh6fq\") pod \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\" (UID: \"5a34ee25-d63f-4ed2-8a45-304105ec08ee\") " Oct 31 01:09:33.144252 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.144228 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a34ee25-d63f-4ed2-8a45-304105ec08ee-plugins-config-volume" pod "5a34ee25-d63f-4ed2-8a45-304105ec08ee" (UID: "5a34ee25-d63f-4ed2-8a45-304105ec08ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:09:33.144377 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.144359 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a34ee25-d63f-4ed2-8a45-304105ec08ee-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:33.146132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.146111 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a34ee25-d63f-4ed2-8a45-304105ec08ee-kube-api-access-fh6fq" pod "5a34ee25-d63f-4ed2-8a45-304105ec08ee" (UID: "5a34ee25-d63f-4ed2-8a45-304105ec08ee"). InnerVolumeSpecName "kube-api-access-fh6fq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:09:33.244954 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.244926 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6fq\" (UniqueName: \"kubernetes.io/projected/5a34ee25-d63f-4ed2-8a45-304105ec08ee-kube-api-access-fh6fq\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:09:33.534460 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.534379 5719 generic.go:356] "Generic (PLEG): container finished" podID="5a34ee25-d63f-4ed2-8a45-304105ec08ee" containerID="bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10" exitCode=0 Oct 31 01:09:33.534618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.534513 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" event={"ID":"5a34ee25-d63f-4ed2-8a45-304105ec08ee","Type":"ContainerDied","Data":"bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10"} Oct 31 01:09:33.534618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.534559 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" event={"ID":"5a34ee25-d63f-4ed2-8a45-304105ec08ee","Type":"ContainerDied","Data":"cc62dded3a5ac6886d4da53ff767dae39ca7e0c4676c14109922285867b5ca32"} Oct 31 01:09:33.534618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.534588 5719 scope.go:122] "RemoveContainer" containerID="bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10" Oct 31 01:09:33.534790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.534627 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf" Oct 31 01:09:33.545537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.545511 5719 scope.go:122] "RemoveContainer" containerID="bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10" Oct 31 01:09:33.545827 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:09:33.545798 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10\": container with ID starting with bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10 not found: ID does not exist" containerID="bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10" Oct 31 01:09:33.545923 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.545840 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10"} err="failed to get container status \"bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10\": rpc error: code = NotFound desc = could not find container \"bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10\": container with ID starting with bdbf128449c70987bd8e1969a45c4a1aea51b5135bb1f06f6959518283638f10 not found: ID does not exist" Oct 31 01:09:33.694167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.694137 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf"] Oct 31 01:09:33.742015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:33.741982 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9e4801-97fbd5b87-h2lzf"] Oct 31 01:09:34.602685 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:34.602653 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a34ee25-d63f-4ed2-8a45-304105ec08ee" path="/var/lib/kubelet/pods/5a34ee25-d63f-4ed2-8a45-304105ec08ee/volumes" Oct 31 01:09:35.113568 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.113535 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7"] Oct 31 01:09:35.114013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.113996 5719 cpu_manager.go:414] "containerMap: removing container" podUID="cf50bc8c-b168-440f-b4fb-e54075233151" containerName="revoke-keys" Oct 31 01:09:35.114097 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114014 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf50bc8c-b168-440f-b4fb-e54075233151" containerName="revoke-keys" Oct 31 01:09:35.114097 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114025 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" Oct 31 01:09:35.114097 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114030 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" Oct 31 01:09:35.114097 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114051 5719 cpu_manager.go:414] "containerMap: removing container" podUID="9f2c6fd9-2445-4f03-a860-c279276fc1cc" containerName="revoke-keys" Oct 31 01:09:35.114097 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114057 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2c6fd9-2445-4f03-a860-c279276fc1cc" containerName="revoke-keys" Oct 31 01:09:35.114097 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114088 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5a34ee25-d63f-4ed2-8a45-304105ec08ee" containerName="payload-processing" Oct 31 01:09:35.114097 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114094 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a34ee25-d63f-4ed2-8a45-304105ec08ee" containerName="payload-processing" Oct 31 01:09:35.114322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114162 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a34ee25-d63f-4ed2-8a45-304105ec08ee" containerName="payload-processing" Oct 31 01:09:35.114322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114171 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf50bc8c-b168-440f-b4fb-e54075233151" containerName="revoke-keys" Oct 31 01:09:35.114322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114178 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5608df70-fa36-4805-8893-1277b632649a" containerName="istio-proxy" Oct 31 01:09:35.114322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.114187 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2c6fd9-2445-4f03-a860-c279276fc1cc" containerName="revoke-keys" Oct 31 01:09:35.118870 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.118848 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:35.162179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.162156 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7"] Oct 31 01:09:35.205597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.205561 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ba34eb\"" Oct 31 01:09:35.205810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.205677 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ba34eb-dockercfg-hfqx9\"" Oct 31 01:09:35.267598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.267570 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls6pd\" (UniqueName: \"kubernetes.io/projected/364f5aab-0dc0-4eef-a6a2-0c3966206372-kube-api-access-ls6pd\") pod \"payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7\" (UID: \"364f5aab-0dc0-4eef-a6a2-0c3966206372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:35.267839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.267818 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364f5aab-0dc0-4eef-a6a2-0c3966206372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7\" (UID: \"364f5aab-0dc0-4eef-a6a2-0c3966206372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:35.277225 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.277196 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv"] Oct 31 01:09:35.282128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.282107 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:35.336932 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.336908 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv"] Oct 31 01:09:35.369258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.369203 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364f5aab-0dc0-4eef-a6a2-0c3966206372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7\" (UID: \"364f5aab-0dc0-4eef-a6a2-0c3966206372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:35.369258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.369236 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c8447-8e55-481f-ac4e-2363cfe5792b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv\" (UID: \"839c8447-8e55-481f-ac4e-2363cfe5792b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:35.369409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.369267 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls6pd\" (UniqueName: \"kubernetes.io/projected/364f5aab-0dc0-4eef-a6a2-0c3966206372-kube-api-access-ls6pd\") pod \"payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7\" (UID: \"364f5aab-0dc0-4eef-a6a2-0c3966206372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:35.369409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.369329 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqlf\" (UniqueName: \"kubernetes.io/projected/839c8447-8e55-481f-ac4e-2363cfe5792b-kube-api-access-6fqlf\") pod \"payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv\" (UID: \"839c8447-8e55-481f-ac4e-2363cfe5792b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:35.369885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.369868 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364f5aab-0dc0-4eef-a6a2-0c3966206372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7\" (UID: \"364f5aab-0dc0-4eef-a6a2-0c3966206372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:35.430512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.430486 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls6pd\" (UniqueName: \"kubernetes.io/projected/364f5aab-0dc0-4eef-a6a2-0c3966206372-kube-api-access-ls6pd\") pod \"payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7\" (UID: \"364f5aab-0dc0-4eef-a6a2-0c3966206372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:35.432350 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.432325 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:35.470380 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.470352 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqlf\" (UniqueName: \"kubernetes.io/projected/839c8447-8e55-481f-ac4e-2363cfe5792b-kube-api-access-6fqlf\") pod \"payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv\" (UID: \"839c8447-8e55-481f-ac4e-2363cfe5792b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:35.470516 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.470502 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c8447-8e55-481f-ac4e-2363cfe5792b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv\" (UID: \"839c8447-8e55-481f-ac4e-2363cfe5792b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:35.471154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.471136 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c8447-8e55-481f-ac4e-2363cfe5792b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv\" (UID: \"839c8447-8e55-481f-ac4e-2363cfe5792b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:35.529595 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.529568 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqlf\" (UniqueName: \"kubernetes.io/projected/839c8447-8e55-481f-ac4e-2363cfe5792b-kube-api-access-6fqlf\") pod \"payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv\" (UID: \"839c8447-8e55-481f-ac4e-2363cfe5792b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:35.595622 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.595598 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:35.805586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.805555 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7"] Oct 31 01:09:35.809530 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:09:35.809505 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364f5aab_0dc0_4eef_a6a2_0c3966206372.slice/crio-640ebd7f049cbea309306d39c80418c2d01c9a0ce9dfd54c5a5f90fb03d5fe08 WatchSource:0}: Error finding container 640ebd7f049cbea309306d39c80418c2d01c9a0ce9dfd54c5a5f90fb03d5fe08: Status 404 returned error can't find the container with id 640ebd7f049cbea309306d39c80418c2d01c9a0ce9dfd54c5a5f90fb03d5fe08 Oct 31 01:09:35.968663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:35.968594 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv"] Oct 31 01:09:35.972682 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:09:35.972659 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839c8447_8e55_481f_ac4e_2363cfe5792b.slice/crio-996953a77bad19773601bc5564d67e7e874c0c2132e95b3dd2b97c17e52800f8 WatchSource:0}: Error finding container 996953a77bad19773601bc5564d67e7e874c0c2132e95b3dd2b97c17e52800f8: Status 404 returned error can't find the container with id 996953a77bad19773601bc5564d67e7e874c0c2132e95b3dd2b97c17e52800f8 Oct 31 01:09:36.052772 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.052746 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z"] Oct 31 01:09:36.059822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.059798 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10f688fd-l298z"] Oct 31 01:09:36.562407 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.562368 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" event={"ID":"839c8447-8e55-481f-ac4e-2363cfe5792b","Type":"ContainerStarted","Data":"23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b"} Oct 31 01:09:36.562597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.562440 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" event={"ID":"839c8447-8e55-481f-ac4e-2363cfe5792b","Type":"ContainerStarted","Data":"996953a77bad19773601bc5564d67e7e874c0c2132e95b3dd2b97c17e52800f8"} Oct 31 01:09:36.562597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.562471 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:36.564148 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.564121 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" event={"ID":"364f5aab-0dc0-4eef-a6a2-0c3966206372","Type":"ContainerStarted","Data":"18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d"} Oct 31 01:09:36.564267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.564164 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" event={"ID":"364f5aab-0dc0-4eef-a6a2-0c3966206372","Type":"ContainerStarted","Data":"640ebd7f049cbea309306d39c80418c2d01c9a0ce9dfd54c5a5f90fb03d5fe08"} Oct 31 01:09:36.564267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.564200 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:36.603361 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.603337 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97054a4a-835c-4bf9-a96b-f8f29726bac5" path="/var/lib/kubelet/pods/97054a4a-835c-4bf9-a96b-f8f29726bac5/volumes" Oct 31 01:09:36.675435 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.675384 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" podStartSLOduration=1.6753661260000001 podStartE2EDuration="1.675366126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:36.673412171 +0000 UTC m=+2458.209554476" watchObservedRunningTime="2026-10-31 01:09:36.675366126 +0000 UTC m=+2458.211508435" Oct 31 01:09:36.789230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:36.789177 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" podStartSLOduration=1.789163366 podStartE2EDuration="1.789163366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:36.786222631 +0000 UTC m=+2458.322364933" watchObservedRunningTime="2026-10-31 01:09:36.789163366 +0000 UTC m=+2458.325305671" Oct 31 01:09:41.190881 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:41.190852 5719 scope.go:122] "RemoveContainer" containerID="4e45196583cf738b0f6d5dd4206b1bb90ddc7d099a3878f7b6a2d2524abe5f5e" Oct 31 01:09:41.200733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:41.200711 5719 scope.go:122] "RemoveContainer" containerID="8536865e872482bbef1d7b914b0a7fd38de67b72c1d146e806783f18db8a987d" Oct 31 01:09:47.574192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:47.574160 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:09:47.574607 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:47.574253 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:09:52.052293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:52.052251 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr"] Oct 31 01:09:52.100604 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:52.100565 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7fbcf9-7qmdr"] Oct 31 01:09:52.605122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:52.605085 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29d6c03-12ba-4687-b4c7-755c60313cac" path="/var/lib/kubelet/pods/b29d6c03-12ba-4687-b4c7-755c60313cac/volumes" Oct 31 01:09:54.385847 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.385811 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq"] Oct 31 01:09:54.423028 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.423003 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9e4801-bd6pq"] Oct 31 01:09:54.604390 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.604358 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf50bc8c-b168-440f-b4fb-e54075233151" path="/var/lib/kubelet/pods/cf50bc8c-b168-440f-b4fb-e54075233151/volumes" Oct 31 01:09:54.614884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.614863 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d"] Oct 31 01:09:54.620648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.620630 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:54.671481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.671414 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4397fc\"" Oct 31 01:09:54.671624 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.671601 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4397fc-dockercfg-2fjh9\"" Oct 31 01:09:54.673793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.673771 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d"] Oct 31 01:09:54.686096 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.686075 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4704dfad-d066-4b5d-a97c-daed41f983f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d\" (UID: \"4704dfad-d066-4b5d-a97c-daed41f983f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:54.686213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.686117 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrvdl\" (UniqueName: \"kubernetes.io/projected/4704dfad-d066-4b5d-a97c-daed41f983f7-kube-api-access-wrvdl\") pod \"payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d\" (UID: \"4704dfad-d066-4b5d-a97c-daed41f983f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:54.761117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.761082 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd"] Oct 31 01:09:54.765155 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.765132 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:54.787680 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.787650 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4704dfad-d066-4b5d-a97c-daed41f983f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d\" (UID: \"4704dfad-d066-4b5d-a97c-daed41f983f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:54.787814 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.787733 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrvdl\" (UniqueName: \"kubernetes.io/projected/4704dfad-d066-4b5d-a97c-daed41f983f7-kube-api-access-wrvdl\") pod \"payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d\" (UID: \"4704dfad-d066-4b5d-a97c-daed41f983f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:54.788327 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.788311 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4704dfad-d066-4b5d-a97c-daed41f983f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d\" (UID: \"4704dfad-d066-4b5d-a97c-daed41f983f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:54.820443 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.820419 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd"] Oct 31 01:09:54.846735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.846709 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrvdl\" (UniqueName: \"kubernetes.io/projected/4704dfad-d066-4b5d-a97c-daed41f983f7-kube-api-access-wrvdl\") pod \"payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d\" (UID: \"4704dfad-d066-4b5d-a97c-daed41f983f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:54.888671 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.888647 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ce7c3-62bb-4abe-b797-465f984d022a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd\" (UID: \"fe5ce7c3-62bb-4abe-b797-465f984d022a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:54.888805 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.888706 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ng9f\" (UniqueName: \"kubernetes.io/projected/fe5ce7c3-62bb-4abe-b797-465f984d022a-kube-api-access-9ng9f\") pod \"payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd\" (UID: \"fe5ce7c3-62bb-4abe-b797-465f984d022a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:54.934204 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.934148 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:54.990509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.990474 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ce7c3-62bb-4abe-b797-465f984d022a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd\" (UID: \"fe5ce7c3-62bb-4abe-b797-465f984d022a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:54.990639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.990557 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ng9f\" (UniqueName: \"kubernetes.io/projected/fe5ce7c3-62bb-4abe-b797-465f984d022a-kube-api-access-9ng9f\") pod \"payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd\" (UID: \"fe5ce7c3-62bb-4abe-b797-465f984d022a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:54.991557 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:54.991515 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ce7c3-62bb-4abe-b797-465f984d022a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd\" (UID: \"fe5ce7c3-62bb-4abe-b797-465f984d022a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:55.051628 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.051603 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ng9f\" (UniqueName: \"kubernetes.io/projected/fe5ce7c3-62bb-4abe-b797-465f984d022a-kube-api-access-9ng9f\") pod \"payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd\" (UID: \"fe5ce7c3-62bb-4abe-b797-465f984d022a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:55.078859 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.078833 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:55.312051 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.311981 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d"] Oct 31 01:09:55.315994 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:09:55.315970 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4704dfad_d066_4b5d_a97c_daed41f983f7.slice/crio-afa02207b9b0813f70ad49eef0a3f8ed5af7fded603da7457dece629e25c3f02 WatchSource:0}: Error finding container afa02207b9b0813f70ad49eef0a3f8ed5af7fded603da7457dece629e25c3f02: Status 404 returned error can't find the container with id afa02207b9b0813f70ad49eef0a3f8ed5af7fded603da7457dece629e25c3f02 Oct 31 01:09:55.453870 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.453845 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd"] Oct 31 01:09:55.458238 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:09:55.458209 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5ce7c3_62bb_4abe_b797_465f984d022a.slice/crio-1fdf8d36a8c5c8c8c39a2a38fb5398d1c8e02ac6b66b81efc7f662e2feca15f1 WatchSource:0}: Error finding container 1fdf8d36a8c5c8c8c39a2a38fb5398d1c8e02ac6b66b81efc7f662e2feca15f1: Status 404 returned error can't find the container with id 1fdf8d36a8c5c8c8c39a2a38fb5398d1c8e02ac6b66b81efc7f662e2feca15f1 Oct 31 01:09:55.710313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.710287 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" event={"ID":"fe5ce7c3-62bb-4abe-b797-465f984d022a","Type":"ContainerStarted","Data":"0d53012c636626005da9e2e25f6b14a96b587227fc67343cf15c574e3c6327f6"} Oct 31 01:09:55.710492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.710337 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" event={"ID":"fe5ce7c3-62bb-4abe-b797-465f984d022a","Type":"ContainerStarted","Data":"1fdf8d36a8c5c8c8c39a2a38fb5398d1c8e02ac6b66b81efc7f662e2feca15f1"} Oct 31 01:09:55.710492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.710451 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:09:55.711979 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.711956 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" event={"ID":"4704dfad-d066-4b5d-a97c-daed41f983f7","Type":"ContainerStarted","Data":"ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493"} Oct 31 01:09:55.712093 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.711991 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" event={"ID":"4704dfad-d066-4b5d-a97c-daed41f983f7","Type":"ContainerStarted","Data":"afa02207b9b0813f70ad49eef0a3f8ed5af7fded603da7457dece629e25c3f02"} Oct 31 01:09:55.712093 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.712020 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:09:55.822891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.822850 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" podStartSLOduration=1.822838399 podStartE2EDuration="1.822838399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:55.821883554 +0000 UTC m=+2477.358025859" watchObservedRunningTime="2026-10-31 01:09:55.822838399 +0000 UTC m=+2477.358980704" Oct 31 01:09:55.944377 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:09:55.944328 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" podStartSLOduration=1.944314786 podStartE2EDuration="1.944314786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:55.941709055 +0000 UTC m=+2477.477851354" watchObservedRunningTime="2026-10-31 01:09:55.944314786 +0000 UTC m=+2477.480457092" Oct 31 01:10:03.009177 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:10:03.009131 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:10:05.276426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:05.276396 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:10:06.721176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:06.721142 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:10:06.721580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:06.721205 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:10:08.053510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:08.053477 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq"] Oct 31 01:10:08.101082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:08.101053 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b3f100-ztmkq"] Oct 31 01:10:08.602940 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:08.602908 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a141e3c-e314-4d2f-a107-ffcaec0b1230" path="/var/lib/kubelet/pods/5a141e3c-e314-4d2f-a107-ffcaec0b1230/volumes" Oct 31 01:10:18.917528 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.917497 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt"] Oct 31 01:10:18.923078 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.923059 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934002 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.933972 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.934025 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.934105 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm589\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-kube-api-access-zm589\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.934161 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.934190 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934337 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.934304 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6edb592-7b44-4e7b-801c-90c632be47a4-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.934377 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.934409 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:18.934592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:18.934477 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.025397 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.025367 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt"] Oct 31 01:10:19.035120 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035082 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035120 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035117 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035136 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6edb592-7b44-4e7b-801c-90c632be47a4-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035152 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035170 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035196 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035310 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035356 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035398 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm589\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-kube-api-access-zm589\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035526 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-credential-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.035618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035597 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-socket\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.036215 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035811 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-certs\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.036215 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.035887 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-data\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.036373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.036347 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6edb592-7b44-4e7b-801c-90c632be47a4-istiod-ca-cert\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.038412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.038391 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-envoy\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.038829 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.038809 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-podinfo\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.094116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.094092 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm589\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-kube-api-access-zm589\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.094250 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.094127 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-token\") pod \"e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.239197 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.239139 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:19.667891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.667861 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt"] Oct 31 01:10:19.671463 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:10:19.671436 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6edb592_7b44_4e7b_801c_90c632be47a4.slice/crio-f2a8bb6c40458de2f7f7628c05c4fa3da7cc2661c74572cd4cb6ed73fa893987 WatchSource:0}: Error finding container f2a8bb6c40458de2f7f7628c05c4fa3da7cc2661c74572cd4cb6ed73fa893987: Status 404 returned error can't find the container with id f2a8bb6c40458de2f7f7628c05c4fa3da7cc2661c74572cd4cb6ed73fa893987 Oct 31 01:10:19.673856 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.673825 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:10:19.673933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.673893 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:10:19.673933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.673922 5719 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:10:19.907506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.907452 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" event={"ID":"d6edb592-7b44-4e7b-801c-90c632be47a4","Type":"ContainerStarted","Data":"18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722"} Oct 31 01:10:19.907768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:19.907524 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" event={"ID":"d6edb592-7b44-4e7b-801c-90c632be47a4","Type":"ContainerStarted","Data":"f2a8bb6c40458de2f7f7628c05c4fa3da7cc2661c74572cd4cb6ed73fa893987"} Oct 31 01:10:20.027387 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:20.027294 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podStartSLOduration=2.027280591 podStartE2EDuration="2.027280591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:20.026417013 +0000 UTC m=+2501.562559344" watchObservedRunningTime="2026-10-31 01:10:20.027280591 +0000 UTC m=+2501.563422896" Oct 31 01:10:20.239661 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:20.239620 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:20.241621 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:20.241594 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 31 01:10:20.241802 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:20.241654 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 31 01:10:21.239461 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:21.239430 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 31 01:10:21.239977 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:21.239487 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 31 01:10:22.239584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:22.239540 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 31 01:10:22.240117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:22.239615 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 31 01:10:23.240423 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:23.240384 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 31 01:10:23.240876 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:23.240450 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 31 01:10:24.239682 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:24.239643 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 31 01:10:24.239893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:24.239730 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 31 01:10:25.050908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:25.050865 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6"] Oct 31 01:10:25.102727 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:25.099719 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0575d4f-swfh6"] Oct 31 01:10:25.240249 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:25.240213 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 31 01:10:25.240432 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:25.240268 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 31 01:10:26.240434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:26.240395 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 31 01:10:26.240800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:26.240454 5719 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 31 01:10:26.605198 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:26.605165 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70762982-ee52-4b72-a7cd-8debe6b74455" path="/var/lib/kubelet/pods/70762982-ee52-4b72-a7cd-8debe6b74455/volumes" Oct 31 01:10:27.244018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:27.243992 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:27.244394 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:27.244362 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:27.245146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:27.245127 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:10:33.009622 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:10:33.009583 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:10:36.048833 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:36.048800 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6"] Oct 31 01:10:36.095999 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:36.095975 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b41a8d-nw4p6"] Oct 31 01:10:36.604742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:36.604686 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a01bf40-450f-4ad4-bb34-3e5c16555ae4" path="/var/lib/kubelet/pods/5a01bf40-450f-4ad4-bb34-3e5c16555ae4/volumes" Oct 31 01:10:41.250766 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:41.250735 5719 scope.go:122] "RemoveContainer" containerID="761037df0e59824e30d58329d8201cce484fd5666cb43161df416360bda648de" Oct 31 01:10:41.260557 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:41.260537 5719 scope.go:122] "RemoveContainer" containerID="4c726103c81409c56cda094ff5edc76bc1c9518cc544911fd25e7418e45951d8" Oct 31 01:10:46.548575 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:46.548493 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:10:52.050511 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:52.050477 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd"] Oct 31 01:10:52.099373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:52.099340 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c325f7f4-ndcjd"] Oct 31 01:10:52.602396 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:10:52.602359 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc14856-e90e-47c6-869e-4f7b07885e6b" path="/var/lib/kubelet/pods/1cc14856-e90e-47c6-869e-4f7b07885e6b/volumes" Oct 31 01:11:03.010086 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:11:03.010043 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:11:03.050152 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:03.050118 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk"] Oct 31 01:11:03.097133 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:03.097103 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86117516-6ndpk"] Oct 31 01:11:04.606883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:04.605609 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa25b9f-0c3a-4684-a48c-46440d264d57" path="/var/lib/kubelet/pods/5fa25b9f-0c3a-4684-a48c-46440d264d57/volumes" Oct 31 01:11:12.434702 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.434640 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782"] Oct 31 01:11:12.439162 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.439141 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:12.488779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.488746 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782"] Oct 31 01:11:12.489834 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.489799 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a72754\"" Oct 31 01:11:12.532603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.532576 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-a72754-dockercfg-fsr5v\"" Oct 31 01:11:12.534887 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.534854 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176128e0-a162-4957-8fc3-aba7a4076be7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782\" (UID: \"176128e0-a162-4957-8fc3-aba7a4076be7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:12.535013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.534916 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz968\" (UniqueName: \"kubernetes.io/projected/176128e0-a162-4957-8fc3-aba7a4076be7-kube-api-access-qz968\") pod \"payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782\" (UID: \"176128e0-a162-4957-8fc3-aba7a4076be7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:12.578980 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.578948 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l"] Oct 31 01:11:12.583063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.583042 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:12.636263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.636230 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz968\" (UniqueName: \"kubernetes.io/projected/176128e0-a162-4957-8fc3-aba7a4076be7-kube-api-access-qz968\") pod \"payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782\" (UID: \"176128e0-a162-4957-8fc3-aba7a4076be7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:12.637000 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.636970 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176128e0-a162-4957-8fc3-aba7a4076be7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782\" (UID: \"176128e0-a162-4957-8fc3-aba7a4076be7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:12.638108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.637984 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176128e0-a162-4957-8fc3-aba7a4076be7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782\" (UID: \"176128e0-a162-4957-8fc3-aba7a4076be7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:12.638262 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.638239 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l"] Oct 31 01:11:12.696476 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.696408 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz968\" (UniqueName: \"kubernetes.io/projected/176128e0-a162-4957-8fc3-aba7a4076be7-kube-api-access-qz968\") pod \"payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782\" (UID: \"176128e0-a162-4957-8fc3-aba7a4076be7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:12.739119 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.739090 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwddg\" (UniqueName: \"kubernetes.io/projected/19a35525-1b77-4772-b089-b8a786ab452b-kube-api-access-fwddg\") pod \"payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l\" (UID: \"19a35525-1b77-4772-b089-b8a786ab452b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:12.739274 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.739150 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a35525-1b77-4772-b089-b8a786ab452b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l\" (UID: \"19a35525-1b77-4772-b089-b8a786ab452b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:12.753568 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.753546 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:12.840509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.840427 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwddg\" (UniqueName: \"kubernetes.io/projected/19a35525-1b77-4772-b089-b8a786ab452b-kube-api-access-fwddg\") pod \"payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l\" (UID: \"19a35525-1b77-4772-b089-b8a786ab452b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:12.840509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.840499 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a35525-1b77-4772-b089-b8a786ab452b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l\" (UID: \"19a35525-1b77-4772-b089-b8a786ab452b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:12.841319 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.841298 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a35525-1b77-4772-b089-b8a786ab452b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l\" (UID: \"19a35525-1b77-4772-b089-b8a786ab452b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:12.900310 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.900280 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwddg\" (UniqueName: \"kubernetes.io/projected/19a35525-1b77-4772-b089-b8a786ab452b-kube-api-access-fwddg\") pod \"payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l\" (UID: \"19a35525-1b77-4772-b089-b8a786ab452b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:12.902377 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:12.902352 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:13.130869 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:13.130830 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782"] Oct 31 01:11:13.135340 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:11:13.135314 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176128e0_a162_4957_8fc3_aba7a4076be7.slice/crio-a224d54985f5599f42ed9c45affeca70bd7828a1e19454a5727664954c106fa8 WatchSource:0}: Error finding container a224d54985f5599f42ed9c45affeca70bd7828a1e19454a5727664954c106fa8: Status 404 returned error can't find the container with id a224d54985f5599f42ed9c45affeca70bd7828a1e19454a5727664954c106fa8 Oct 31 01:11:13.280211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:13.280138 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l"] Oct 31 01:11:13.285324 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:11:13.285296 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a35525_1b77_4772_b089_b8a786ab452b.slice/crio-9368c24872ef7e36a1d66dc701017472425181b53eb67478d2ec1b4c46a56a01 WatchSource:0}: Error finding container 9368c24872ef7e36a1d66dc701017472425181b53eb67478d2ec1b4c46a56a01: Status 404 returned error can't find the container with id 9368c24872ef7e36a1d66dc701017472425181b53eb67478d2ec1b4c46a56a01 Oct 31 01:11:13.293558 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:13.293526 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" event={"ID":"19a35525-1b77-4772-b089-b8a786ab452b","Type":"ContainerStarted","Data":"9368c24872ef7e36a1d66dc701017472425181b53eb67478d2ec1b4c46a56a01"} Oct 31 01:11:13.295058 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:13.295035 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" event={"ID":"176128e0-a162-4957-8fc3-aba7a4076be7","Type":"ContainerStarted","Data":"02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232"} Oct 31 01:11:13.295154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:13.295075 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" event={"ID":"176128e0-a162-4957-8fc3-aba7a4076be7","Type":"ContainerStarted","Data":"a224d54985f5599f42ed9c45affeca70bd7828a1e19454a5727664954c106fa8"} Oct 31 01:11:13.295191 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:13.295146 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:13.412710 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:13.412642 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" podStartSLOduration=1.412628144 podStartE2EDuration="1.412628144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:13.410298321 +0000 UTC m=+2554.946440639" watchObservedRunningTime="2026-10-31 01:11:13.412628144 +0000 UTC m=+2554.948770443" Oct 31 01:11:14.302528 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:14.302491 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" event={"ID":"19a35525-1b77-4772-b089-b8a786ab452b","Type":"ContainerStarted","Data":"b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29"} Oct 31 01:11:14.302965 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:14.302557 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:14.417556 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:14.417504 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" podStartSLOduration=2.417490248 podStartE2EDuration="2.417490248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:14.415565659 +0000 UTC m=+2555.951707966" watchObservedRunningTime="2026-10-31 01:11:14.417490248 +0000 UTC m=+2555.953632552" Oct 31 01:11:20.067023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:20.066992 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f"] Oct 31 01:11:20.114576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:20.114546 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-682cb3fc-gfs4f"] Oct 31 01:11:20.602116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:20.602081 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1210ae8-451c-4438-9e07-135e30dbb333" path="/var/lib/kubelet/pods/d1210ae8-451c-4438-9e07-135e30dbb333/volumes" Oct 31 01:11:24.305817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:24.305790 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:11:25.312189 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:25.312150 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:11:32.709361 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:32.709324 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:11:33.010766 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:11:33.010659 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:11:38.779902 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:38.779857 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s"] Oct 31 01:11:38.783913 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:38.783894 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:38.835751 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:38.835525 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-3be6fa\"" Oct 31 01:11:38.886087 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:38.886054 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s"] Oct 31 01:11:38.935482 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:38.935438 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s\" (UID: \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:38.935870 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:38.935845 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbghm\" (UniqueName: \"kubernetes.io/projected/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-kube-api-access-xbghm\") pod \"payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s\" (UID: \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:38.964231 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:38.964195 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf"] Oct 31 01:11:38.968878 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:38.968856 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:39.019365 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.019288 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf"] Oct 31 01:11:39.019866 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.019843 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-3be6fa-dockercfg-x9kvh\"" Oct 31 01:11:39.037831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.037754 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e30dbf8-db51-4772-9f60-852d38b8d182-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf\" (UID: \"9e30dbf8-db51-4772-9f60-852d38b8d182\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:39.037831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.037793 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s\" (UID: \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:39.038070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.037884 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7b4h\" (UniqueName: \"kubernetes.io/projected/9e30dbf8-db51-4772-9f60-852d38b8d182-kube-api-access-w7b4h\") pod \"payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf\" (UID: \"9e30dbf8-db51-4772-9f60-852d38b8d182\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:39.038070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.037920 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbghm\" (UniqueName: \"kubernetes.io/projected/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-kube-api-access-xbghm\") pod \"payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s\" (UID: \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:39.038604 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.038583 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s\" (UID: \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:39.096901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.096867 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbghm\" (UniqueName: \"kubernetes.io/projected/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-kube-api-access-xbghm\") pod \"payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s\" (UID: \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:39.098722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.098682 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:39.138778 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.138739 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e30dbf8-db51-4772-9f60-852d38b8d182-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf\" (UID: \"9e30dbf8-db51-4772-9f60-852d38b8d182\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:39.138941 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.138823 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7b4h\" (UniqueName: \"kubernetes.io/projected/9e30dbf8-db51-4772-9f60-852d38b8d182-kube-api-access-w7b4h\") pod \"payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf\" (UID: \"9e30dbf8-db51-4772-9f60-852d38b8d182\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:39.139439 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.139413 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e30dbf8-db51-4772-9f60-852d38b8d182-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf\" (UID: \"9e30dbf8-db51-4772-9f60-852d38b8d182\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:39.199444 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.199415 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7b4h\" (UniqueName: \"kubernetes.io/projected/9e30dbf8-db51-4772-9f60-852d38b8d182-kube-api-access-w7b4h\") pod \"payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf\" (UID: \"9e30dbf8-db51-4772-9f60-852d38b8d182\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:39.284929 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.284896 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:39.486419 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.486334 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s"] Oct 31 01:11:39.489429 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:11:39.489402 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f4bbb5_318a_43d4_81e2_e907fb9b9b0b.slice/crio-09cd18a1e87d88326a887362035802f37ca449421f717e9ab7d57cba9e49ac85 WatchSource:0}: Error finding container 09cd18a1e87d88326a887362035802f37ca449421f717e9ab7d57cba9e49ac85: Status 404 returned error can't find the container with id 09cd18a1e87d88326a887362035802f37ca449421f717e9ab7d57cba9e49ac85 Oct 31 01:11:39.661196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:39.661160 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf"] Oct 31 01:11:39.665197 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:11:39.665168 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e30dbf8_db51_4772_9f60_852d38b8d182.slice/crio-781c7a0f1ae919743d855654da82ffeafeec245dd71b566f92ff29aab70eff52 WatchSource:0}: Error finding container 781c7a0f1ae919743d855654da82ffeafeec245dd71b566f92ff29aab70eff52: Status 404 returned error can't find the container with id 781c7a0f1ae919743d855654da82ffeafeec245dd71b566f92ff29aab70eff52 Oct 31 01:11:40.493398 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:40.493358 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" event={"ID":"9e30dbf8-db51-4772-9f60-852d38b8d182","Type":"ContainerStarted","Data":"647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510"} Oct 31 01:11:40.493833 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:40.493419 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" event={"ID":"9e30dbf8-db51-4772-9f60-852d38b8d182","Type":"ContainerStarted","Data":"781c7a0f1ae919743d855654da82ffeafeec245dd71b566f92ff29aab70eff52"} Oct 31 01:11:40.493833 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:40.493447 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:40.495222 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:40.495196 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" event={"ID":"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b","Type":"ContainerStarted","Data":"95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386"} Oct 31 01:11:40.495347 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:40.495239 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" event={"ID":"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b","Type":"ContainerStarted","Data":"09cd18a1e87d88326a887362035802f37ca449421f717e9ab7d57cba9e49ac85"} Oct 31 01:11:40.495347 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:40.495279 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:40.608850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:40.608780 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" podStartSLOduration=2.6087679599999998 podStartE2EDuration="2.60876796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:40.60600945 +0000 UTC m=+2582.142151754" watchObservedRunningTime="2026-10-31 01:11:40.60876796 +0000 UTC m=+2582.144910293" Oct 31 01:11:40.718827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:40.718769 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" podStartSLOduration=2.718754013 podStartE2EDuration="2.718754013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:40.717661943 +0000 UTC m=+2582.253804249" watchObservedRunningTime="2026-10-31 01:11:40.718754013 +0000 UTC m=+2582.254896317" Oct 31 01:11:41.295463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:41.295434 5719 scope.go:122] "RemoveContainer" containerID="4ee6f8f4168d0dc045f09a6993ce0da2accf930daf94deb8ed0c2c8a5998d4ca" Oct 31 01:11:41.305958 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:41.305932 5719 scope.go:122] "RemoveContainer" containerID="73de2777b92366a8d87d81986c6472f7d04daf7788c40d886a82f2c1f3504596" Oct 31 01:11:41.315057 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:41.315039 5719 scope.go:122] "RemoveContainer" containerID="7ee9476cb710b0cd73100b6b724eaafa6cdd5b267d8639f81f4151bc3ce4581f" Oct 31 01:11:41.326551 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:41.326526 5719 scope.go:122] "RemoveContainer" containerID="bb1e83eb03d230ddae5e753b1dca2144cce71fa5fce5878c0eec0a3aedaffba9" Oct 31 01:11:41.335652 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:41.335635 5719 scope.go:122] "RemoveContainer" containerID="38880015a7ad5ab9c867486e79d9035c2129def746044a0efabe090910210601" Oct 31 01:11:43.051353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:43.051320 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr"] Oct 31 01:11:43.099382 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:43.099338 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40626f1b-bg8pr"] Oct 31 01:11:44.604596 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:44.604552 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d" path="/var/lib/kubelet/pods/69cd0b66-ddf4-43e0-8d72-963bd5ae1f5d/volumes" Oct 31 01:11:51.505311 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:51.505282 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:11:51.505896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:51.505350 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:11:57.050495 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:57.050458 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv"] Oct 31 01:11:57.095760 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:57.095729 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eed20ea9-gpbqv"] Oct 31 01:11:58.603606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:58.603575 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5" path="/var/lib/kubelet/pods/b354f7f8-d2da-48b3-88f5-4dfd40dbbcd5/volumes" Oct 31 01:11:59.112852 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.112814 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt"] Oct 31 01:11:59.124268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.124235 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:11:59.137144 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.137115 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k"] Oct 31 01:11:59.141319 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.141303 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:11:59.169529 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.169498 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt"] Oct 31 01:11:59.178328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.178297 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt\" (UID: \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:11:59.178484 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.178410 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6fv\" (UniqueName: \"kubernetes.io/projected/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-kube-api-access-pz6fv\") pod \"payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt\" (UID: \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:11:59.181870 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.181850 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f9925a\"" Oct 31 01:11:59.181968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.181865 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f9925a-dockercfg-2cskk\"" Oct 31 01:11:59.216896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.216868 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k"] Oct 31 01:11:59.281138 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.281098 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6fv\" (UniqueName: \"kubernetes.io/projected/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-kube-api-access-pz6fv\") pod \"payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt\" (UID: \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:11:59.281344 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.281255 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f707cc-b82a-44a3-907d-d71336fa75db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k\" (UID: \"12f707cc-b82a-44a3-907d-d71336fa75db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:11:59.281344 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.281291 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxlwd\" (UniqueName: \"kubernetes.io/projected/12f707cc-b82a-44a3-907d-d71336fa75db-kube-api-access-gxlwd\") pod \"payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k\" (UID: \"12f707cc-b82a-44a3-907d-d71336fa75db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:11:59.281469 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.281435 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt\" (UID: \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:11:59.282198 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.282174 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt\" (UID: \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:11:59.342202 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.342165 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6fv\" (UniqueName: \"kubernetes.io/projected/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-kube-api-access-pz6fv\") pod \"payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt\" (UID: \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:11:59.383018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.382915 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f707cc-b82a-44a3-907d-d71336fa75db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k\" (UID: \"12f707cc-b82a-44a3-907d-d71336fa75db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:11:59.383018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.382968 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxlwd\" (UniqueName: \"kubernetes.io/projected/12f707cc-b82a-44a3-907d-d71336fa75db-kube-api-access-gxlwd\") pod \"payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k\" (UID: \"12f707cc-b82a-44a3-907d-d71336fa75db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:11:59.383767 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.383747 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f707cc-b82a-44a3-907d-d71336fa75db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k\" (UID: \"12f707cc-b82a-44a3-907d-d71336fa75db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:11:59.438609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.438570 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:11:59.442494 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.442474 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxlwd\" (UniqueName: \"kubernetes.io/projected/12f707cc-b82a-44a3-907d-d71336fa75db-kube-api-access-gxlwd\") pod \"payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k\" (UID: \"12f707cc-b82a-44a3-907d-d71336fa75db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:11:59.454503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.454469 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:11:59.822755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.822725 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt"] Oct 31 01:11:59.826527 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:11:59.826492 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0186e230_3fc0_4c2a_b9c3_18da6fa83a1a.slice/crio-12c4fe65788be24496b69973e3aaa770d75aa732eb34f4cc3c117f4f5f3bc679 WatchSource:0}: Error finding container 12c4fe65788be24496b69973e3aaa770d75aa732eb34f4cc3c117f4f5f3bc679: Status 404 returned error can't find the container with id 12c4fe65788be24496b69973e3aaa770d75aa732eb34f4cc3c117f4f5f3bc679 Oct 31 01:11:59.874559 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:11:59.874532 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k"] Oct 31 01:11:59.876918 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:11:59.876892 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f707cc_b82a_44a3_907d_d71336fa75db.slice/crio-49201749d2202f17a33284a5451abb594709c0edbe4d004be6ff900d4e4bd0c3 WatchSource:0}: Error finding container 49201749d2202f17a33284a5451abb594709c0edbe4d004be6ff900d4e4bd0c3: Status 404 returned error can't find the container with id 49201749d2202f17a33284a5451abb594709c0edbe4d004be6ff900d4e4bd0c3 Oct 31 01:12:00.635765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:00.635731 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" event={"ID":"12f707cc-b82a-44a3-907d-d71336fa75db","Type":"ContainerStarted","Data":"248a060829df7164635d40e12ccc802ca57722bcbb9a926d2db57d6317478618"} Oct 31 01:12:00.635983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:00.635789 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" event={"ID":"12f707cc-b82a-44a3-907d-d71336fa75db","Type":"ContainerStarted","Data":"49201749d2202f17a33284a5451abb594709c0edbe4d004be6ff900d4e4bd0c3"} Oct 31 01:12:00.635983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:00.635818 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:12:00.637383 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:00.637361 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" event={"ID":"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a","Type":"ContainerStarted","Data":"123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c"} Oct 31 01:12:00.637482 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:00.637400 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" event={"ID":"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a","Type":"ContainerStarted","Data":"12c4fe65788be24496b69973e3aaa770d75aa732eb34f4cc3c117f4f5f3bc679"} Oct 31 01:12:00.637518 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:00.637491 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:12:00.754717 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:00.754647 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" podStartSLOduration=1.754633616 podStartE2EDuration="1.754633616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:00.754205344 +0000 UTC m=+2602.290347648" watchObservedRunningTime="2026-10-31 01:12:00.754633616 +0000 UTC m=+2602.290775974" Oct 31 01:12:00.869747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:00.869669 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" podStartSLOduration=1.869654476 podStartE2EDuration="1.869654476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:00.866171438 +0000 UTC m=+2602.402313743" watchObservedRunningTime="2026-10-31 01:12:00.869654476 +0000 UTC m=+2602.405796782" Oct 31 01:12:03.010716 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:12:03.010667 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:12:11.647656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:11.647628 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:12:11.648154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:11.647726 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:12:14.049893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:14.049854 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst"] Oct 31 01:12:14.098215 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:14.098179 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e47d5cd5-xgmst"] Oct 31 01:12:14.604335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:14.604304 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20143222-43f7-4b4e-b1a5-6c0c91038c17" path="/var/lib/kubelet/pods/20143222-43f7-4b4e-b1a5-6c0c91038c17/volumes" Oct 31 01:12:26.048825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:26.048744 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9"] Oct 31 01:12:26.096052 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:26.096017 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99c9f0dd-hr5p9"] Oct 31 01:12:26.604344 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:26.604315 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81561031-f6f9-48f6-874b-ca096d06ae90" path="/var/lib/kubelet/pods/81561031-f6f9-48f6-874b-ca096d06ae90/volumes" Oct 31 01:12:31.828137 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:31.828103 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:12:33.016923 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:12:33.016766 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:12:41.235408 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.235370 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg"] Oct 31 01:12:41.239209 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.239189 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:41.338104 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.338078 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-7f8596\"" Oct 31 01:12:41.338252 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.338176 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-7f8596-dockercfg-dfw68\"" Oct 31 01:12:41.339687 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.339666 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg\" (UID: \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:41.339937 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.339921 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldxc\" (UniqueName: \"kubernetes.io/projected/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-kube-api-access-wldxc\") pod \"payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg\" (UID: \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:41.341085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.341064 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg"] Oct 31 01:12:41.377718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.377672 5719 scope.go:122] "RemoveContainer" containerID="7417d083d95c5f03553716e707b985db9c42d7798d7d34a96e1972a2bd31e054" Oct 31 01:12:41.392284 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.392158 5719 scope.go:122] "RemoveContainer" containerID="85318a549f2871ee1267298b6c7dcedaebf5fcc849835525215e92b637a285cd" Oct 31 01:12:41.401996 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.401959 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz"] Oct 31 01:12:41.405666 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.405645 5719 scope.go:122] "RemoveContainer" containerID="247adeeb9281f7835ab0115f9ec70a82a8d518babc8cb9ea5f9df709b19dea73" Oct 31 01:12:41.408179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.408158 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:41.424530 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.424494 5719 scope.go:122] "RemoveContainer" containerID="f859837c36f5a874ed5431445869afdb2c302cdd9463a5084faa9e119fa3adca" Oct 31 01:12:41.437851 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.437834 5719 scope.go:122] "RemoveContainer" containerID="5c61033f6ab46c30e3856da48b9fcdd5525407e675c7a3001a79a512983b3714" Oct 31 01:12:41.441355 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.441333 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldxc\" (UniqueName: \"kubernetes.io/projected/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-kube-api-access-wldxc\") pod \"payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg\" (UID: \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:41.441447 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.441373 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg\" (UID: \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:41.442067 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.442046 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg\" (UID: \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:41.447054 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.447036 5719 scope.go:122] "RemoveContainer" containerID="ea6cd56caf034ba42e35003da822e7cb88dda4ef184faecb480ae0112956c8d3" Oct 31 01:12:41.456947 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.456921 5719 scope.go:122] "RemoveContainer" containerID="57e3f6f1a93373167645b2ac47dcbfaeab82445b5dd9fefd26200c4302e79e6e" Oct 31 01:12:41.461377 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.461354 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz"] Oct 31 01:12:41.503151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.502939 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldxc\" (UniqueName: \"kubernetes.io/projected/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-kube-api-access-wldxc\") pod \"payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg\" (UID: \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:41.543062 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.543039 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz\" (UID: \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:41.543189 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.543121 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwqj\" (UniqueName: \"kubernetes.io/projected/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-kube-api-access-4cwqj\") pod \"payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz\" (UID: \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:41.553352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.553334 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:41.644436 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.644388 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwqj\" (UniqueName: \"kubernetes.io/projected/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-kube-api-access-4cwqj\") pod \"payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz\" (UID: \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:41.644644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.644614 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz\" (UID: \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:41.645827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.645774 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz\" (UID: \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:41.704091 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.704068 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwqj\" (UniqueName: \"kubernetes.io/projected/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-kube-api-access-4cwqj\") pod \"payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz\" (UID: \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:41.734944 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.734921 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:41.930984 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:41.930957 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg"] Oct 31 01:12:41.935294 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:12:41.935267 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ec5e5c_0313_4b7d_a8bf_bac04731c7d8.slice/crio-2fa49c8a4356a46cd00ad831698e1caaaef196d93ce394286c5d53be3d8bea4d WatchSource:0}: Error finding container 2fa49c8a4356a46cd00ad831698e1caaaef196d93ce394286c5d53be3d8bea4d: Status 404 returned error can't find the container with id 2fa49c8a4356a46cd00ad831698e1caaaef196d93ce394286c5d53be3d8bea4d Oct 31 01:12:42.129708 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:42.129667 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz"] Oct 31 01:12:42.131658 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:12:42.131631 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9e6b31_7248_4f5a_8805_2da63bc3ab4b.slice/crio-4bab30505daf2017e9a2653525cca3ecb1604892bfd9b151aea46080f7656663 WatchSource:0}: Error finding container 4bab30505daf2017e9a2653525cca3ecb1604892bfd9b151aea46080f7656663: Status 404 returned error can't find the container with id 4bab30505daf2017e9a2653525cca3ecb1604892bfd9b151aea46080f7656663 Oct 31 01:12:42.931366 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:42.931324 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" event={"ID":"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8","Type":"ContainerStarted","Data":"1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72"} Oct 31 01:12:42.931822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:42.931392 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:42.931822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:42.931421 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" event={"ID":"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8","Type":"ContainerStarted","Data":"2fa49c8a4356a46cd00ad831698e1caaaef196d93ce394286c5d53be3d8bea4d"} Oct 31 01:12:42.933022 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:42.932999 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" event={"ID":"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b","Type":"ContainerStarted","Data":"46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729"} Oct 31 01:12:42.933149 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:42.933041 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" event={"ID":"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b","Type":"ContainerStarted","Data":"4bab30505daf2017e9a2653525cca3ecb1604892bfd9b151aea46080f7656663"} Oct 31 01:12:42.933149 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:42.933076 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:43.052151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:43.052097 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" podStartSLOduration=2.052080847 podStartE2EDuration="2.052080847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:43.047061788 +0000 UTC m=+2644.583204093" watchObservedRunningTime="2026-10-31 01:12:43.052080847 +0000 UTC m=+2644.588223150" Oct 31 01:12:43.164678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:43.164631 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" podStartSLOduration=2.164618509 podStartE2EDuration="2.164618509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:43.162376836 +0000 UTC m=+2644.698519141" watchObservedRunningTime="2026-10-31 01:12:43.164618509 +0000 UTC m=+2644.700760829" Oct 31 01:12:50.056758 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:50.056721 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd"] Oct 31 01:12:50.104205 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:50.104173 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52eebf9e-2xnzd"] Oct 31 01:12:50.603065 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:50.603032 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0204857-cb05-414f-9d9c-f122774df276" path="/var/lib/kubelet/pods/a0204857-cb05-414f-9d9c-f122774df276/volumes" Oct 31 01:12:53.941548 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:53.941521 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:12:53.941962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:53.941709 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:12:57.050346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:57.050312 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl"] Oct 31 01:12:57.095894 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:57.095866 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d5b65d-njqnl"] Oct 31 01:12:58.605055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:12:58.605023 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f42a8e4-015d-478f-818b-95313d5cc9fe" path="/var/lib/kubelet/pods/2f42a8e4-015d-478f-818b-95313d5cc9fe/volumes" Oct 31 01:13:01.213896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:01.213859 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:13:03.009114 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:13:03.009078 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:13:05.049059 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:05.049025 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96"] Oct 31 01:13:05.051664 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:05.051632 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03dd63dc-xtv96"] Oct 31 01:13:06.602815 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:06.602782 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5826ee17-9007-4e30-a02c-9701137c8757" path="/var/lib/kubelet/pods/5826ee17-9007-4e30-a02c-9701137c8757/volumes" Oct 31 01:13:13.048226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:13.048192 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57"] Oct 31 01:13:13.096956 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:13.096922 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e200a174-lxm57"] Oct 31 01:13:14.604716 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:14.604668 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64053c11-c2fd-4b5d-9728-7a6647a555cf" path="/var/lib/kubelet/pods/64053c11-c2fd-4b5d-9728-7a6647a555cf/volumes" Oct 31 01:13:21.047370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:21.047336 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb"] Oct 31 01:13:21.095041 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:21.095012 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44aa87d-h2vlb"] Oct 31 01:13:22.604040 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:22.604005 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43de3169-70bf-4f38-9c9d-9ea88eda15af" path="/var/lib/kubelet/pods/43de3169-70bf-4f38-9c9d-9ea88eda15af/volumes" Oct 31 01:13:29.801286 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:29.801255 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:13:33.010049 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:13:33.010017 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:13:37.052867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:37.052820 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5"] Oct 31 01:13:37.100308 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:37.100277 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007bea65-twns5"] Oct 31 01:13:38.467404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.467369 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd"] Oct 31 01:13:38.471341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.471322 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.531981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.531953 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdt5j\" (UniqueName: \"kubernetes.io/projected/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-kube-api-access-kdt5j\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.532113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.532004 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.532161 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.532130 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.569126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.569094 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:13:38.569126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.569105 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:13:38.569338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.569222 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:38.569394 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.569339 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:13:38.571932 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.571911 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd"] Oct 31 01:13:38.603866 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.603844 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2bcb6f-d7da-4e7e-92b2-9817ab886d88" path="/var/lib/kubelet/pods/7b2bcb6f-d7da-4e7e-92b2-9817ab886d88/volumes" Oct 31 01:13:38.632795 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.632768 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdt5j\" (UniqueName: \"kubernetes.io/projected/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-kube-api-access-kdt5j\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.632899 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.632821 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.632899 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.632879 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.681966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.681928 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:38.682062 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.682038 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:13:38.683495 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.683473 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.685227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.685207 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.738488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.738428 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:13:38.741785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.741763 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdt5j\" (UniqueName: \"kubernetes.io/projected/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-kube-api-access-kdt5j\") pod \"maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:38.835162 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.835137 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:13:38.836171 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:38.836156 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:39.212102 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:39.212068 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd"] Oct 31 01:13:39.216883 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:13:39.216853 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32a98f3_ebf0_466b_a4b7_29c2a7cfe909.slice/crio-7ade1b61b6d2d971eee57b42af299c2b9e6023c47ba1ed5371c9edf87f4eae95 WatchSource:0}: Error finding container 7ade1b61b6d2d971eee57b42af299c2b9e6023c47ba1ed5371c9edf87f4eae95: Status 404 returned error can't find the container with id 7ade1b61b6d2d971eee57b42af299c2b9e6023c47ba1ed5371c9edf87f4eae95 Oct 31 01:13:39.316449 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:39.316423 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" event={"ID":"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909","Type":"ContainerStarted","Data":"7ade1b61b6d2d971eee57b42af299c2b9e6023c47ba1ed5371c9edf87f4eae95"} Oct 31 01:13:40.325411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:40.325373 5719 generic.go:356] "Generic (PLEG): container finished" podID="f32a98f3-ebf0-466b-a4b7-29c2a7cfe909" containerID="223d1eef5e2155d049b8c1f4f0c4213b43d7defc3f27df577825cb679abb60b1" exitCode=0 Oct 31 01:13:40.325881 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:40.325465 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" event={"ID":"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909","Type":"ContainerDied","Data":"223d1eef5e2155d049b8c1f4f0c4213b43d7defc3f27df577825cb679abb60b1"} Oct 31 01:13:40.967306 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:40.967273 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc"] Oct 31 01:13:40.971276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:40.971245 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.027123 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.027097 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc"] Oct 31 01:13:41.056559 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.056526 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tmp-dir\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.056685 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.056575 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-home\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.056747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.056726 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-dshm\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.056789 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.056774 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27flj\" (UniqueName: \"kubernetes.io/projected/a68090ac-329e-4b98-a6b4-77fb3cd7095a-kube-api-access-27flj\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.056850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.056833 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-model-cache\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.056914 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.056897 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tls-certs\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.067463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.067445 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-8fb7f0\"/\"openshift-service-ca.crt\"" Oct 31 01:13:41.071251 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.071214 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-8fb7f0\"/\"default-dockercfg-rfxxj\"" Oct 31 01:13:41.071467 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.071451 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-8fb7f0\"/\"auth-test-model-8fb7f0-kserve-self-signed-certs\"" Oct 31 01:13:41.071535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.071452 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-8fb7f0\"/\"kube-root-ca.crt\"" Oct 31 01:13:41.157911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.157878 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27flj\" (UniqueName: \"kubernetes.io/projected/a68090ac-329e-4b98-a6b4-77fb3cd7095a-kube-api-access-27flj\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.158086 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.157916 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-model-cache\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.158086 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.157949 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tls-certs\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.158086 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.158020 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tmp-dir\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.158086 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.158070 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-home\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.158280 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.158213 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-dshm\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.158340 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.158287 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-model-cache\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.158402 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.158389 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tmp-dir\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.158517 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.158499 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-home\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.160422 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.160401 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-dshm\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.160663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.160646 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tls-certs\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.216935 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.216903 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27flj\" (UniqueName: \"kubernetes.io/projected/a68090ac-329e-4b98-a6b4-77fb3cd7095a-kube-api-access-27flj\") pod \"auth-test-model-8fb7f0-kserve-868495f8d5-s2drc\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.283553 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.283448 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:41.513475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.513449 5719 scope.go:122] "RemoveContainer" containerID="a9cc6ca624b32fbeba0800abae8f9858d170c15ac083da02b49c581b10696d46" Oct 31 01:13:41.523232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.523208 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:41.525655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.525635 5719 scope.go:122] "RemoveContainer" containerID="ac7c3480ba52ffd30da7bd2d868dab7ea60357bf1ccefc543d45e167ce6b3987" Oct 31 01:13:41.536339 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.536319 5719 scope.go:122] "RemoveContainer" containerID="d2282e4a59785e74a1b7a836fa4a7c21d211bd5ebc2712cbad9d2b351cdb16c3" Oct 31 01:13:41.564432 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.564405 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-xks-ca\") pod \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " Oct 31 01:13:41.564593 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.564571 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-kube-api-access-kdt5j\" (UniqueName: \"kubernetes.io/projected/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-kube-api-access-kdt5j\") pod \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " Oct 31 01:13:41.564731 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.564714 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-openshift-ca\") pod \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\" (UID: \"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909\") " Oct 31 01:13:41.565178 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.565153 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-openshift-ca" pod "f32a98f3-ebf0-466b-a4b7-29c2a7cfe909" (UID: "f32a98f3-ebf0-466b-a4b7-29c2a7cfe909"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:41.566512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.566494 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-xks-ca" pod "f32a98f3-ebf0-466b-a4b7-29c2a7cfe909" (UID: "f32a98f3-ebf0-466b-a4b7-29c2a7cfe909"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:41.567104 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.567081 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-kube-api-access-kdt5j" pod "f32a98f3-ebf0-466b-a4b7-29c2a7cfe909" (UID: "f32a98f3-ebf0-466b-a4b7-29c2a7cfe909"). InnerVolumeSpecName "kube-api-access-kdt5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:41.661850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.661822 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc"] Oct 31 01:13:41.666228 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.666206 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:41.666325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.666230 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdt5j\" (UniqueName: \"kubernetes.io/projected/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-kube-api-access-kdt5j\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:41.666325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:41.666246 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:13:42.342605 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:42.342579 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" Oct 31 01:13:42.342822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:42.342578 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd" event={"ID":"f32a98f3-ebf0-466b-a4b7-29c2a7cfe909","Type":"ContainerDied","Data":"7ade1b61b6d2d971eee57b42af299c2b9e6023c47ba1ed5371c9edf87f4eae95"} Oct 31 01:13:42.342822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:42.342776 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ade1b61b6d2d971eee57b42af299c2b9e6023c47ba1ed5371c9edf87f4eae95" Oct 31 01:13:42.344548 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:42.344520 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" event={"ID":"a68090ac-329e-4b98-a6b4-77fb3cd7095a","Type":"ContainerStarted","Data":"f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648"} Oct 31 01:13:42.344665 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:42.344560 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" event={"ID":"a68090ac-329e-4b98-a6b4-77fb3cd7095a","Type":"ContainerStarted","Data":"64ab3339eea85d904da8fa48633a715c67c75ae74ace398ca3679c9c4bced8d3"} Oct 31 01:13:42.457851 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:42.457782 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" podStartSLOduration=2.457771007 podStartE2EDuration="2.457771007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:42.45667352 +0000 UTC m=+2703.992815824" watchObservedRunningTime="2026-10-31 01:13:42.457771007 +0000 UTC m=+2703.993913312" Oct 31 01:13:43.284220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:43.284182 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:45.048277 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:45.048240 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq"] Oct 31 01:13:45.097177 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:45.097144 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3770b5-8xzsq"] Oct 31 01:13:46.603790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:46.603675 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164c731b-0862-4031-9a2f-7c08752ca78a" path="/var/lib/kubelet/pods/164c731b-0862-4031-9a2f-7c08752ca78a/volumes" Oct 31 01:13:50.049034 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:50.049008 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh"] Oct 31 01:13:50.097990 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:50.097964 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e0c7894-86rzh"] Oct 31 01:13:50.602730 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:50.602684 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468a6d5c-bb0b-4442-9ac9-9a90ef20eb67" path="/var/lib/kubelet/pods/468a6d5c-bb0b-4442-9ac9-9a90ef20eb67/volumes" Oct 31 01:13:51.284042 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:51.284006 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:51.296608 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:51.296584 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:13:51.412012 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:13:51.411985 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:14:02.050506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:02.050474 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5"] Oct 31 01:14:02.097750 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:02.097684 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5789ca9-fm9d5"] Oct 31 01:14:02.100523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:02.100499 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7"] Oct 31 01:14:02.102597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:02.102576 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb285d7-mvxk7"] Oct 31 01:14:02.605187 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:02.605156 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e932d2c-21ce-48d2-87c1-98c9b64d64a0" path="/var/lib/kubelet/pods/8e932d2c-21ce-48d2-87c1-98c9b64d64a0/volumes" Oct 31 01:14:02.605738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:02.605719 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abcaccce-b4bf-4db4-bae2-8c3b585ebeac" path="/var/lib/kubelet/pods/abcaccce-b4bf-4db4-bae2-8c3b585ebeac/volumes" Oct 31 01:14:03.011448 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:14:03.011374 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:14:03.848560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:03.848531 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:14:19.047733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:19.047703 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk"] Oct 31 01:14:19.095098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:19.095065 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c83aa7b5-rlqlk"] Oct 31 01:14:20.603734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:20.603703 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2c6fd9-2445-4f03-a860-c279276fc1cc" path="/var/lib/kubelet/pods/9f2c6fd9-2445-4f03-a860-c279276fc1cc/volumes" Oct 31 01:14:33.010750 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:14:33.010714 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:14:41.580559 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.580528 5719 scope.go:122] "RemoveContainer" containerID="a773f5df1b53b5e03c88395ff72df561a190077312e55f97557c241931e33896" Oct 31 01:14:41.592085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.592042 5719 scope.go:122] "RemoveContainer" containerID="274aebdfeb325ac3645cb631bf9f4b87c2379b3b4a7aa29717ee5128bd80993d" Oct 31 01:14:41.602288 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.602263 5719 scope.go:122] "RemoveContainer" containerID="dbb9bd2cbfae98f7cada1a959fba8bbd697a09db19d4447def3d20e772c368a5" Oct 31 01:14:41.614230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.614214 5719 scope.go:122] "RemoveContainer" containerID="beee0b47520dda070cb8b957fe34942c93f9007d2db2a21958dc3fde5a5e933b" Oct 31 01:14:41.623107 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.623083 5719 scope.go:122] "RemoveContainer" containerID="f9fe7f2a1b3161dac430206eb337961e38316e20bc805c6ce8eeb542ab71258d" Oct 31 01:14:41.631129 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.631113 5719 scope.go:122] "RemoveContainer" containerID="d42282a9cef4a9e6fb35746851ec20871a8fcff0812bb1899565285a7a146394" Oct 31 01:14:41.639233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.639216 5719 scope.go:122] "RemoveContainer" containerID="525affda30217642fe8abf39c9202be2d3f99335c8ba15611c8d972f01a4beb6" Oct 31 01:14:41.647246 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.647229 5719 scope.go:122] "RemoveContainer" containerID="2ab3da2ee92f45d891e64b68f6b70686fdff3e7d17eba631526e1d5509c7f3a3" Oct 31 01:14:41.655107 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.655074 5719 scope.go:122] "RemoveContainer" containerID="71e54d4ba03abda88b88f9b46db043dd327e8a0740443b1b4ce4b8e348043703" Oct 31 01:14:41.810749 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.810723 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9"] Oct 31 01:14:41.811359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.811343 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f32a98f3-ebf0-466b-a4b7-29c2a7cfe909" containerName="revoke-keys" Oct 31 01:14:41.811412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.811361 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32a98f3-ebf0-466b-a4b7-29c2a7cfe909" containerName="revoke-keys" Oct 31 01:14:41.811471 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.811462 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32a98f3-ebf0-466b-a4b7-29c2a7cfe909" containerName="revoke-keys" Oct 31 01:14:41.816445 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.816429 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.891578 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.891499 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tmp-dir\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.891578 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.891532 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-model-cache\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.891846 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.891674 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tls-certs\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.891846 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.891833 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-home\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.891971 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.891866 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc52j\" (UniqueName: \"kubernetes.io/projected/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-kube-api-access-vc52j\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.892029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.891989 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-dshm\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.914787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.914759 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a72754\"/\"openshift-service-ca.crt\"" Oct 31 01:14:41.914787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.914769 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-a72754\"/\"kube-root-ca.crt\"" Oct 31 01:14:41.914972 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.914800 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a72754\"/\"auth-test-model-a72754-kserve-self-signed-certs\"" Oct 31 01:14:41.914972 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.914829 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-a72754\"/\"default-dockercfg-vvfrt\"" Oct 31 01:14:41.917342 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.917324 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9"] Oct 31 01:14:41.993113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.993074 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tmp-dir\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.993113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.993127 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-model-cache\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.993455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.993255 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tls-certs\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.993537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.993482 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-home\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.993537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.993514 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc52j\" (UniqueName: \"kubernetes.io/projected/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-kube-api-access-vc52j\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.993650 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.993584 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-dshm\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.993733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.993682 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tmp-dir\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.994024 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.993996 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-model-cache\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.994229 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.994032 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-home\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.996177 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.996149 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tls-certs\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:41.996326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:41.996308 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-dshm\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:42.053527 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:42.053494 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc52j\" (UniqueName: \"kubernetes.io/projected/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-kube-api-access-vc52j\") pod \"auth-test-model-a72754-kserve-7f4556749c-k24t9\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:42.129299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:42.129276 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:42.517444 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:42.517373 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9"] Oct 31 01:14:42.522685 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:14:42.522659 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61a03f47_6bc3_4449_b7b6_fcbebf2b3bae.slice/crio-bd0b6985c040cdb8d67ab5684e1d043bb6a23edbc521ea5ee97c72b6474feae9 WatchSource:0}: Error finding container bd0b6985c040cdb8d67ab5684e1d043bb6a23edbc521ea5ee97c72b6474feae9: Status 404 returned error can't find the container with id bd0b6985c040cdb8d67ab5684e1d043bb6a23edbc521ea5ee97c72b6474feae9 Oct 31 01:14:42.738420 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:42.738384 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" event={"ID":"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae","Type":"ContainerStarted","Data":"4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76"} Oct 31 01:14:42.738817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:42.738429 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" event={"ID":"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae","Type":"ContainerStarted","Data":"bd0b6985c040cdb8d67ab5684e1d043bb6a23edbc521ea5ee97c72b6474feae9"} Oct 31 01:14:42.851188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:42.851145 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" podStartSLOduration=1.851132383 podStartE2EDuration="1.851132383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:42.850855659 +0000 UTC m=+2764.386997964" watchObservedRunningTime="2026-10-31 01:14:42.851132383 +0000 UTC m=+2764.387274687" Oct 31 01:14:43.130008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:43.129912 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:43.721758 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:43.721724 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:14:52.129687 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:52.129650 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:52.142258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:52.142234 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:52.822399 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:52.822373 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:14:58.931007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:58.930977 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd"] Oct 31 01:14:58.978362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:14:58.978334 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f1d081-f9whd"] Oct 31 01:15:00.125147 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.125106 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn"] Oct 31 01:15:00.125655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.125392 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" containerID="cri-o://6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e" gracePeriod=30 Oct 31 01:15:00.209786 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.209754 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2"] Oct 31 01:15:00.214176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.214154 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6"] Oct 31 01:15:00.214338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.214274 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:00.218864 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.218849 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:00.238423 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.238403 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9"] Oct 31 01:15:00.242298 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.242283 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72"] Oct 31 01:15:00.242429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.242411 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:00.246726 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.246670 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw"] Oct 31 01:15:00.246840 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.246793 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:00.253862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.253841 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg"] Oct 31 01:15:00.254110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.254055 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:00.257839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.257819 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd"] Oct 31 01:15:00.257954 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.257940 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:00.262312 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.262292 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx"] Oct 31 01:15:00.262706 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.262671 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:00.266983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.266965 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:00.282543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.282522 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:15:00.282656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.282605 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:15:00.282656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.282627 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:15:00.286682 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.286652 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657"] Oct 31 01:15:00.288589 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.288562 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2\" (UID: \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:00.288677 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.288653 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4tz2\" (UniqueName: \"kubernetes.io/projected/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-kube-api-access-v4tz2\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2\" (UID: \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:00.288780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.288708 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjn5b\" (UniqueName: \"kubernetes.io/projected/e94b5461-808b-4892-9573-2d81fc28db7c-kube-api-access-hjn5b\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6\" (UID: \"e94b5461-808b-4892-9573-2d81fc28db7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:00.288848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.288817 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94b5461-808b-4892-9573-2d81fc28db7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6\" (UID: \"e94b5461-808b-4892-9573-2d81fc28db7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:00.291672 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.291476 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h"] Oct 31 01:15:00.291672 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.291580 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:00.299819 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.299798 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2"] Oct 31 01:15:00.300222 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.299842 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6"] Oct 31 01:15:00.300222 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.299930 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:00.302085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.302062 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9"] Oct 31 01:15:00.312607 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.312580 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw"] Oct 31 01:15:00.332517 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.332495 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx"] Oct 31 01:15:00.334801 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.334780 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg"] Oct 31 01:15:00.343879 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.343742 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657"] Oct 31 01:15:00.360087 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.360050 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h"] Oct 31 01:15:00.365415 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.362560 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72"] Oct 31 01:15:00.367865 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.367842 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd"] Oct 31 01:15:00.390304 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390243 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2\" (UID: \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:00.390430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390317 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqxt\" (UniqueName: \"kubernetes.io/projected/e89e07dd-bed4-4675-adc8-ea689e709b98-kube-api-access-sbqxt\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9\" (UID: \"e89e07dd-bed4-4675-adc8-ea689e709b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:00.390430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390351 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqf67\" (UniqueName: \"kubernetes.io/projected/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-kube-api-access-lqf67\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg\" (UID: \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:00.390430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390374 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbl6r\" (UniqueName: \"kubernetes.io/projected/8b798301-db3c-4a7f-81f2-528d286dc5fb-kube-api-access-xbl6r\") pod \"maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx\" (UID: \"8b798301-db3c-4a7f-81f2-528d286dc5fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:00.390430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390406 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4tz2\" (UniqueName: \"kubernetes.io/projected/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-kube-api-access-v4tz2\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2\" (UID: \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:00.390663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390431 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg\" (UID: \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:00.390663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390464 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjn5b\" (UniqueName: \"kubernetes.io/projected/e94b5461-808b-4892-9573-2d81fc28db7c-kube-api-access-hjn5b\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6\" (UID: \"e94b5461-808b-4892-9573-2d81fc28db7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:00.390663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390490 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e07dd-bed4-4675-adc8-ea689e709b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9\" (UID: \"e89e07dd-bed4-4675-adc8-ea689e709b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:00.390663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390522 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b798301-db3c-4a7f-81f2-528d286dc5fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx\" (UID: \"8b798301-db3c-4a7f-81f2-528d286dc5fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:00.390663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390572 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0b818-27c4-4a4c-b593-e8bbf5051650-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw\" (UID: \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:00.390663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390605 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707c53af-0fef-46f9-a204-a315e0933fe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h\" (UID: \"707c53af-0fef-46f9-a204-a315e0933fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:00.390663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390636 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9798576-77d5-4c89-90ba-2c092992010b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72\" (UID: \"f9798576-77d5-4c89-90ba-2c092992010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:00.391070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390663 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e53823-39bb-4b96-8677-efd71615f28f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657\" (UID: \"65e53823-39bb-4b96-8677-efd71615f28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:00.391070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390739 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94b5461-808b-4892-9573-2d81fc28db7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6\" (UID: \"e94b5461-808b-4892-9573-2d81fc28db7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:00.391070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390825 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dll\" (UniqueName: \"kubernetes.io/projected/707c53af-0fef-46f9-a204-a315e0933fe6-kube-api-access-l2dll\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h\" (UID: \"707c53af-0fef-46f9-a204-a315e0933fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:00.391070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390856 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x7ch\" (UniqueName: \"kubernetes.io/projected/0cd0b818-27c4-4a4c-b593-e8bbf5051650-kube-api-access-5x7ch\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw\" (UID: \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:00.391070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.390951 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnj62\" (UniqueName: \"kubernetes.io/projected/f9798576-77d5-4c89-90ba-2c092992010b-kube-api-access-nnj62\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72\" (UID: \"f9798576-77d5-4c89-90ba-2c092992010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:00.391070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.391007 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjsm\" (UniqueName: \"kubernetes.io/projected/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-kube-api-access-pwjsm\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd\" (UID: \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:00.391070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.391067 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmvk\" (UniqueName: \"kubernetes.io/projected/65e53823-39bb-4b96-8677-efd71615f28f-kube-api-access-9kmvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657\" (UID: \"65e53823-39bb-4b96-8677-efd71615f28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:00.391445 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.391110 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd\" (UID: \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:00.391823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.391801 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2\" (UID: \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:00.392773 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.392752 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94b5461-808b-4892-9573-2d81fc28db7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6\" (UID: \"e94b5461-808b-4892-9573-2d81fc28db7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:00.452627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.452604 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4tz2\" (UniqueName: \"kubernetes.io/projected/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-kube-api-access-v4tz2\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2\" (UID: \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:00.452790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.452604 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjn5b\" (UniqueName: \"kubernetes.io/projected/e94b5461-808b-4892-9573-2d81fc28db7c-kube-api-access-hjn5b\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6\" (UID: \"e94b5461-808b-4892-9573-2d81fc28db7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:00.491931 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.491912 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmvk\" (UniqueName: \"kubernetes.io/projected/65e53823-39bb-4b96-8677-efd71615f28f-kube-api-access-9kmvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657\" (UID: \"65e53823-39bb-4b96-8677-efd71615f28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:00.492046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.491944 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd\" (UID: \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:00.492046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492007 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqxt\" (UniqueName: \"kubernetes.io/projected/e89e07dd-bed4-4675-adc8-ea689e709b98-kube-api-access-sbqxt\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9\" (UID: \"e89e07dd-bed4-4675-adc8-ea689e709b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:00.492046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492026 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqf67\" (UniqueName: \"kubernetes.io/projected/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-kube-api-access-lqf67\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg\" (UID: \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:00.492046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492041 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbl6r\" (UniqueName: \"kubernetes.io/projected/8b798301-db3c-4a7f-81f2-528d286dc5fb-kube-api-access-xbl6r\") pod \"maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx\" (UID: \"8b798301-db3c-4a7f-81f2-528d286dc5fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:00.492275 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492063 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg\" (UID: \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:00.492328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492287 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e07dd-bed4-4675-adc8-ea689e709b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9\" (UID: \"e89e07dd-bed4-4675-adc8-ea689e709b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:00.492374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492331 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b798301-db3c-4a7f-81f2-528d286dc5fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx\" (UID: \"8b798301-db3c-4a7f-81f2-528d286dc5fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:00.492418 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492382 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0b818-27c4-4a4c-b593-e8bbf5051650-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw\" (UID: \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:00.492524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492412 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707c53af-0fef-46f9-a204-a315e0933fe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h\" (UID: \"707c53af-0fef-46f9-a204-a315e0933fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:00.492524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492442 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9798576-77d5-4c89-90ba-2c092992010b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72\" (UID: \"f9798576-77d5-4c89-90ba-2c092992010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:00.492524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492468 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e53823-39bb-4b96-8677-efd71615f28f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657\" (UID: \"65e53823-39bb-4b96-8677-efd71615f28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:00.492710 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492543 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dll\" (UniqueName: \"kubernetes.io/projected/707c53af-0fef-46f9-a204-a315e0933fe6-kube-api-access-l2dll\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h\" (UID: \"707c53af-0fef-46f9-a204-a315e0933fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:00.492710 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492571 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x7ch\" (UniqueName: \"kubernetes.io/projected/0cd0b818-27c4-4a4c-b593-e8bbf5051650-kube-api-access-5x7ch\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw\" (UID: \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:00.492710 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492654 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj62\" (UniqueName: \"kubernetes.io/projected/f9798576-77d5-4c89-90ba-2c092992010b-kube-api-access-nnj62\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72\" (UID: \"f9798576-77d5-4c89-90ba-2c092992010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:00.492877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492723 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjsm\" (UniqueName: \"kubernetes.io/projected/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-kube-api-access-pwjsm\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd\" (UID: \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:00.492877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492776 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd\" (UID: \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:00.492877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.492840 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg\" (UID: \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:00.493047 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.493006 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e07dd-bed4-4675-adc8-ea689e709b98-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9\" (UID: \"e89e07dd-bed4-4675-adc8-ea689e709b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:00.493209 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.493186 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b798301-db3c-4a7f-81f2-528d286dc5fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx\" (UID: \"8b798301-db3c-4a7f-81f2-528d286dc5fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:00.493268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.493252 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0b818-27c4-4a4c-b593-e8bbf5051650-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw\" (UID: \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:00.493485 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.493462 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e53823-39bb-4b96-8677-efd71615f28f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657\" (UID: \"65e53823-39bb-4b96-8677-efd71615f28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:00.493528 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.493514 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707c53af-0fef-46f9-a204-a315e0933fe6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h\" (UID: \"707c53af-0fef-46f9-a204-a315e0933fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:00.493674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.493659 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9798576-77d5-4c89-90ba-2c092992010b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72\" (UID: \"f9798576-77d5-4c89-90ba-2c092992010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:00.531289 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.531267 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:00.538520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.538504 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:00.555465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.555413 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjsm\" (UniqueName: \"kubernetes.io/projected/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-kube-api-access-pwjsm\") pod \"maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd\" (UID: \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:00.555568 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.555513 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqxt\" (UniqueName: \"kubernetes.io/projected/e89e07dd-bed4-4675-adc8-ea689e709b98-kube-api-access-sbqxt\") pod \"maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9\" (UID: \"e89e07dd-bed4-4675-adc8-ea689e709b98\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:00.555878 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.555857 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dll\" (UniqueName: \"kubernetes.io/projected/707c53af-0fef-46f9-a204-a315e0933fe6-kube-api-access-l2dll\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h\" (UID: \"707c53af-0fef-46f9-a204-a315e0933fe6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:00.556274 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.556252 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqf67\" (UniqueName: \"kubernetes.io/projected/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-kube-api-access-lqf67\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg\" (UID: \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:00.559144 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.559093 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmvk\" (UniqueName: \"kubernetes.io/projected/65e53823-39bb-4b96-8677-efd71615f28f-kube-api-access-9kmvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657\" (UID: \"65e53823-39bb-4b96-8677-efd71615f28f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:00.559892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.559865 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnj62\" (UniqueName: \"kubernetes.io/projected/f9798576-77d5-4c89-90ba-2c092992010b-kube-api-access-nnj62\") pod \"maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72\" (UID: \"f9798576-77d5-4c89-90ba-2c092992010b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:00.560109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.560090 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x7ch\" (UniqueName: \"kubernetes.io/projected/0cd0b818-27c4-4a4c-b593-e8bbf5051650-kube-api-access-5x7ch\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw\" (UID: \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:00.561663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.561646 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:00.562124 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.562089 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbl6r\" (UniqueName: \"kubernetes.io/projected/8b798301-db3c-4a7f-81f2-528d286dc5fb-kube-api-access-xbl6r\") pod \"maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx\" (UID: \"8b798301-db3c-4a7f-81f2-528d286dc5fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:00.573221 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.572964 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:00.580218 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.580194 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:00.594235 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.590646 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:00.613041 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.612839 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:00.631248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.631217 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32a98f3-ebf0-466b-a4b7-29c2a7cfe909" path="/var/lib/kubelet/pods/f32a98f3-ebf0-466b-a4b7-29c2a7cfe909/volumes" Oct 31 01:15:00.645427 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.637818 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:00.645427 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.639983 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:00.656939 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.654130 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:00.935982 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.935953 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6"] Oct 31 01:15:00.938184 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:00.938161 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94b5461_808b_4892_9573_2d81fc28db7c.slice/crio-00e861d349251fad6e836f255a3a383ecfb0fc7f7b0c2f80b5292fd6eec1230b WatchSource:0}: Error finding container 00e861d349251fad6e836f255a3a383ecfb0fc7f7b0c2f80b5292fd6eec1230b: Status 404 returned error can't find the container with id 00e861d349251fad6e836f255a3a383ecfb0fc7f7b0c2f80b5292fd6eec1230b Oct 31 01:15:00.982565 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.982537 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2"] Oct 31 01:15:00.990177 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:00.989445 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c5bca9_c67f_4ab0_845b_bd4f4fad77a2.slice/crio-b2ac684b4565ef68f13fab1e17bb343297c111b8aca5ae93a08118a774c62989 WatchSource:0}: Error finding container b2ac684b4565ef68f13fab1e17bb343297c111b8aca5ae93a08118a774c62989: Status 404 returned error can't find the container with id b2ac684b4565ef68f13fab1e17bb343297c111b8aca5ae93a08118a774c62989 Oct 31 01:15:00.992404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:00.992383 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9"] Oct 31 01:15:00.993671 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:00.993642 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89e07dd_bed4_4675_adc8_ea689e709b98.slice/crio-858181c169219b5d078045e0a960be65ad2f7880b9310c49dbeaae188b84aaa7 WatchSource:0}: Error finding container 858181c169219b5d078045e0a960be65ad2f7880b9310c49dbeaae188b84aaa7: Status 404 returned error can't find the container with id 858181c169219b5d078045e0a960be65ad2f7880b9310c49dbeaae188b84aaa7 Oct 31 01:15:01.070536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.070508 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg"] Oct 31 01:15:01.073360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.073300 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72"] Oct 31 01:15:01.076256 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:01.076226 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9798576_77d5_4c89_90ba_2c092992010b.slice/crio-2ed7871f208a827c9c6dee650aad0cd92ffd73b321c7d699a674f77121b9edfc WatchSource:0}: Error finding container 2ed7871f208a827c9c6dee650aad0cd92ffd73b321c7d699a674f77121b9edfc: Status 404 returned error can't find the container with id 2ed7871f208a827c9c6dee650aad0cd92ffd73b321c7d699a674f77121b9edfc Oct 31 01:15:01.077109 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:01.077084 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659c95e5_6d46_4d75_a6c5_ba0e9be38f0b.slice/crio-eb7b87fdd1d8514bbadc404cf2489d1497042a8e09a02bd03a42b18ef05dcb7a WatchSource:0}: Error finding container eb7b87fdd1d8514bbadc404cf2489d1497042a8e09a02bd03a42b18ef05dcb7a: Status 404 returned error can't find the container with id eb7b87fdd1d8514bbadc404cf2489d1497042a8e09a02bd03a42b18ef05dcb7a Oct 31 01:15:01.077223 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.077205 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd"] Oct 31 01:15:01.082134 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:01.082113 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8017edc8_ca0b_4ac0_aff4_86d78ef3ea89.slice/crio-fb88a0e96ff9b5b261369d47ec44310349ba1ec5d85d5bd43095b1f9f5b7822f WatchSource:0}: Error finding container fb88a0e96ff9b5b261369d47ec44310349ba1ec5d85d5bd43095b1f9f5b7822f: Status 404 returned error can't find the container with id fb88a0e96ff9b5b261369d47ec44310349ba1ec5d85d5bd43095b1f9f5b7822f Oct 31 01:15:01.115020 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.114998 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx"] Oct 31 01:15:01.216995 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:01.216968 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b798301_db3c_4a7f_81f2_528d286dc5fb.slice/crio-12235a13e2f917c9df3ae0ee3b7717c02e1fae9fa0cd2c6e794fd8417536b4a5 WatchSource:0}: Error finding container 12235a13e2f917c9df3ae0ee3b7717c02e1fae9fa0cd2c6e794fd8417536b4a5: Status 404 returned error can't find the container with id 12235a13e2f917c9df3ae0ee3b7717c02e1fae9fa0cd2c6e794fd8417536b4a5 Oct 31 01:15:01.298836 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.298812 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw"] Oct 31 01:15:01.307514 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.307490 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h"] Oct 31 01:15:01.309207 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:01.309178 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd0b818_27c4_4a4c_b593_e8bbf5051650.slice/crio-307ccf09c0cda41c9e0f29709a9eb2ec2344eada254274862e999c82008d4679 WatchSource:0}: Error finding container 307ccf09c0cda41c9e0f29709a9eb2ec2344eada254274862e999c82008d4679: Status 404 returned error can't find the container with id 307ccf09c0cda41c9e0f29709a9eb2ec2344eada254274862e999c82008d4679 Oct 31 01:15:01.405325 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:01.405282 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707c53af_0fef_46f9_a204_a315e0933fe6.slice/crio-4933a40784bceb26604ed9fb423f971222c281e584b5a9709e3017c12ab8b0a6 WatchSource:0}: Error finding container 4933a40784bceb26604ed9fb423f971222c281e584b5a9709e3017c12ab8b0a6: Status 404 returned error can't find the container with id 4933a40784bceb26604ed9fb423f971222c281e584b5a9709e3017c12ab8b0a6 Oct 31 01:15:01.428121 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.428093 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657"] Oct 31 01:15:01.505769 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:01.505142 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e53823_39bb_4b96_8677_efd71615f28f.slice/crio-81348314a8c55383e06cc43a259f0e30e1ea76cf9fd11c38a7031f73755322c2 WatchSource:0}: Error finding container 81348314a8c55383e06cc43a259f0e30e1ea76cf9fd11c38a7031f73755322c2: Status 404 returned error can't find the container with id 81348314a8c55383e06cc43a259f0e30e1ea76cf9fd11c38a7031f73755322c2 Oct 31 01:15:01.873415 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.873388 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" event={"ID":"65e53823-39bb-4b96-8677-efd71615f28f","Type":"ContainerStarted","Data":"81348314a8c55383e06cc43a259f0e30e1ea76cf9fd11c38a7031f73755322c2"} Oct 31 01:15:01.875078 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.875036 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" event={"ID":"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89","Type":"ContainerStarted","Data":"7d862098d9dec41b2cec68498bbc219548034aa7a1a5548279f9c363e1ec203a"} Oct 31 01:15:01.875195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.875107 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" event={"ID":"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89","Type":"ContainerStarted","Data":"fb88a0e96ff9b5b261369d47ec44310349ba1ec5d85d5bd43095b1f9f5b7822f"} Oct 31 01:15:01.876400 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.876377 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" event={"ID":"0cd0b818-27c4-4a4c-b593-e8bbf5051650","Type":"ContainerStarted","Data":"307ccf09c0cda41c9e0f29709a9eb2ec2344eada254274862e999c82008d4679"} Oct 31 01:15:01.877953 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.877933 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" event={"ID":"f9798576-77d5-4c89-90ba-2c092992010b","Type":"ContainerStarted","Data":"39c3b2ab2cc2e342d3a20ea2fc770dcfa7b03ee0530d55fc34c2777844a67eb0"} Oct 31 01:15:01.878063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.877968 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" event={"ID":"f9798576-77d5-4c89-90ba-2c092992010b","Type":"ContainerStarted","Data":"2ed7871f208a827c9c6dee650aad0cd92ffd73b321c7d699a674f77121b9edfc"} Oct 31 01:15:01.879538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.879507 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" event={"ID":"e89e07dd-bed4-4675-adc8-ea689e709b98","Type":"ContainerStarted","Data":"e8beb80f9057ce78a850571904b90ea63090cd179c16307caa16a66d9c82ea68"} Oct 31 01:15:01.879645 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.879556 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" event={"ID":"e89e07dd-bed4-4675-adc8-ea689e709b98","Type":"ContainerStarted","Data":"858181c169219b5d078045e0a960be65ad2f7880b9310c49dbeaae188b84aaa7"} Oct 31 01:15:01.880676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.880657 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" event={"ID":"707c53af-0fef-46f9-a204-a315e0933fe6","Type":"ContainerStarted","Data":"4933a40784bceb26604ed9fb423f971222c281e584b5a9709e3017c12ab8b0a6"} Oct 31 01:15:01.882150 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.882132 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" event={"ID":"8b798301-db3c-4a7f-81f2-528d286dc5fb","Type":"ContainerStarted","Data":"71f6f5c05d6938c8fad076fa4e11322b61c432441a204506e9247d695208170d"} Oct 31 01:15:01.882246 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.882168 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" event={"ID":"8b798301-db3c-4a7f-81f2-528d286dc5fb","Type":"ContainerStarted","Data":"12235a13e2f917c9df3ae0ee3b7717c02e1fae9fa0cd2c6e794fd8417536b4a5"} Oct 31 01:15:01.883611 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.883593 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" event={"ID":"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2","Type":"ContainerStarted","Data":"078380c6160a87541ce66a56337fdbc4f172c37c2855e61b8e58294d967a621b"} Oct 31 01:15:01.883704 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.883636 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" event={"ID":"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2","Type":"ContainerStarted","Data":"b2ac684b4565ef68f13fab1e17bb343297c111b8aca5ae93a08118a774c62989"} Oct 31 01:15:01.885149 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.885123 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" event={"ID":"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b","Type":"ContainerStarted","Data":"3c1e9337ba596b9893be936fabae82e021eb0e8ae7f60b82064a64d38e65e71d"} Oct 31 01:15:01.885221 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.885164 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" event={"ID":"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b","Type":"ContainerStarted","Data":"eb7b87fdd1d8514bbadc404cf2489d1497042a8e09a02bd03a42b18ef05dcb7a"} Oct 31 01:15:01.886641 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.886618 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" event={"ID":"e94b5461-808b-4892-9573-2d81fc28db7c","Type":"ContainerStarted","Data":"37b3bdbf729815f42604abf922b24b167cacb5db03540a04100c6360a0cf6fca"} Oct 31 01:15:01.886764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:01.886664 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" event={"ID":"e94b5461-808b-4892-9573-2d81fc28db7c","Type":"ContainerStarted","Data":"00e861d349251fad6e836f255a3a383ecfb0fc7f7b0c2f80b5292fd6eec1230b"} Oct 31 01:15:02.012265 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.012219 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" podStartSLOduration=2.012206486 podStartE2EDuration="2.012206486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.010473945 +0000 UTC m=+2783.546616249" watchObservedRunningTime="2026-10-31 01:15:02.012206486 +0000 UTC m=+2783.548348826" Oct 31 01:15:02.126846 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.126801 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" podStartSLOduration=2.126785011 podStartE2EDuration="2.126785011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.124668049 +0000 UTC m=+2783.660810352" watchObservedRunningTime="2026-10-31 01:15:02.126785011 +0000 UTC m=+2783.662927309" Oct 31 01:15:02.242393 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.242348 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" podStartSLOduration=2.242333678 podStartE2EDuration="2.242333678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.24191728 +0000 UTC m=+2783.778059586" watchObservedRunningTime="2026-10-31 01:15:02.242333678 +0000 UTC m=+2783.778475983" Oct 31 01:15:02.368509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.368423 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" podStartSLOduration=2.368410073 podStartE2EDuration="2.368410073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.366304154 +0000 UTC m=+2783.902446458" watchObservedRunningTime="2026-10-31 01:15:02.368410073 +0000 UTC m=+2783.904552377" Oct 31 01:15:02.479717 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.479651 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" podStartSLOduration=2.479636623 podStartE2EDuration="2.479636623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.477745563 +0000 UTC m=+2784.013887864" watchObservedRunningTime="2026-10-31 01:15:02.479636623 +0000 UTC m=+2784.015778925" Oct 31 01:15:02.594952 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.594902 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" podStartSLOduration=2.594882114 podStartE2EDuration="2.594882114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.591003533 +0000 UTC m=+2784.127145839" watchObservedRunningTime="2026-10-31 01:15:02.594882114 +0000 UTC m=+2784.131024421" Oct 31 01:15:02.705180 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.705137 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" podStartSLOduration=2.705124634 podStartE2EDuration="2.705124634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:02.704766771 +0000 UTC m=+2784.240909075" watchObservedRunningTime="2026-10-31 01:15:02.705124634 +0000 UTC m=+2784.241266939" Oct 31 01:15:02.895961 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.895925 5719 generic.go:356] "Generic (PLEG): container finished" podID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerID="e8beb80f9057ce78a850571904b90ea63090cd179c16307caa16a66d9c82ea68" exitCode=60 Oct 31 01:15:02.896112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.896014 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" event={"ID":"e89e07dd-bed4-4675-adc8-ea689e709b98","Type":"ContainerDied","Data":"e8beb80f9057ce78a850571904b90ea63090cd179c16307caa16a66d9c82ea68"} Oct 31 01:15:02.896267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.896248 5719 scope.go:122] "RemoveContainer" containerID="e8beb80f9057ce78a850571904b90ea63090cd179c16307caa16a66d9c82ea68" Oct 31 01:15:02.897788 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.897759 5719 generic.go:356] "Generic (PLEG): container finished" podID="707c53af-0fef-46f9-a204-a315e0933fe6" containerID="d300d9d79c7008ec5d8b2dce92a8542fd1eda7ca8098bac60e72c15a44ae5ac7" exitCode=60 Oct 31 01:15:02.897913 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.897854 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" event={"ID":"707c53af-0fef-46f9-a204-a315e0933fe6","Type":"ContainerDied","Data":"d300d9d79c7008ec5d8b2dce92a8542fd1eda7ca8098bac60e72c15a44ae5ac7"} Oct 31 01:15:02.898396 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.898019 5719 scope.go:122] "RemoveContainer" containerID="d300d9d79c7008ec5d8b2dce92a8542fd1eda7ca8098bac60e72c15a44ae5ac7" Oct 31 01:15:02.899965 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.899950 5719 generic.go:356] "Generic (PLEG): container finished" podID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerID="71f6f5c05d6938c8fad076fa4e11322b61c432441a204506e9247d695208170d" exitCode=60 Oct 31 01:15:02.900030 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.900018 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" event={"ID":"8b798301-db3c-4a7f-81f2-528d286dc5fb","Type":"ContainerDied","Data":"71f6f5c05d6938c8fad076fa4e11322b61c432441a204506e9247d695208170d"} Oct 31 01:15:02.900266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.900250 5719 scope.go:122] "RemoveContainer" containerID="71f6f5c05d6938c8fad076fa4e11322b61c432441a204506e9247d695208170d" Oct 31 01:15:02.902109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.902026 5719 generic.go:356] "Generic (PLEG): container finished" podID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerID="078380c6160a87541ce66a56337fdbc4f172c37c2855e61b8e58294d967a621b" exitCode=60 Oct 31 01:15:02.902109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.902073 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" event={"ID":"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2","Type":"ContainerDied","Data":"078380c6160a87541ce66a56337fdbc4f172c37c2855e61b8e58294d967a621b"} Oct 31 01:15:02.902326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.902310 5719 scope.go:122] "RemoveContainer" containerID="078380c6160a87541ce66a56337fdbc4f172c37c2855e61b8e58294d967a621b" Oct 31 01:15:02.903916 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.903890 5719 generic.go:356] "Generic (PLEG): container finished" podID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerID="3c1e9337ba596b9893be936fabae82e021eb0e8ae7f60b82064a64d38e65e71d" exitCode=60 Oct 31 01:15:02.904017 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.903986 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" event={"ID":"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b","Type":"ContainerDied","Data":"3c1e9337ba596b9893be936fabae82e021eb0e8ae7f60b82064a64d38e65e71d"} Oct 31 01:15:02.904236 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.904221 5719 scope.go:122] "RemoveContainer" containerID="3c1e9337ba596b9893be936fabae82e021eb0e8ae7f60b82064a64d38e65e71d" Oct 31 01:15:02.905707 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.905667 5719 generic.go:356] "Generic (PLEG): container finished" podID="e94b5461-808b-4892-9573-2d81fc28db7c" containerID="37b3bdbf729815f42604abf922b24b167cacb5db03540a04100c6360a0cf6fca" exitCode=60 Oct 31 01:15:02.905861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.905845 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" event={"ID":"e94b5461-808b-4892-9573-2d81fc28db7c","Type":"ContainerDied","Data":"37b3bdbf729815f42604abf922b24b167cacb5db03540a04100c6360a0cf6fca"} Oct 31 01:15:02.906119 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.906104 5719 scope.go:122] "RemoveContainer" containerID="37b3bdbf729815f42604abf922b24b167cacb5db03540a04100c6360a0cf6fca" Oct 31 01:15:02.908031 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.908010 5719 generic.go:356] "Generic (PLEG): container finished" podID="65e53823-39bb-4b96-8677-efd71615f28f" containerID="55287e64a12d89d6b40a1699e7049747001ef5abd6debe7e4e455cb16a8c0960" exitCode=60 Oct 31 01:15:02.908126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.908105 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" event={"ID":"65e53823-39bb-4b96-8677-efd71615f28f","Type":"ContainerDied","Data":"55287e64a12d89d6b40a1699e7049747001ef5abd6debe7e4e455cb16a8c0960"} Oct 31 01:15:02.908458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.908438 5719 scope.go:122] "RemoveContainer" containerID="55287e64a12d89d6b40a1699e7049747001ef5abd6debe7e4e455cb16a8c0960" Oct 31 01:15:02.910014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.909921 5719 generic.go:356] "Generic (PLEG): container finished" podID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerID="7d862098d9dec41b2cec68498bbc219548034aa7a1a5548279f9c363e1ec203a" exitCode=60 Oct 31 01:15:02.910014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.909969 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" event={"ID":"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89","Type":"ContainerDied","Data":"7d862098d9dec41b2cec68498bbc219548034aa7a1a5548279f9c363e1ec203a"} Oct 31 01:15:02.910315 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.910289 5719 scope.go:122] "RemoveContainer" containerID="7d862098d9dec41b2cec68498bbc219548034aa7a1a5548279f9c363e1ec203a" Oct 31 01:15:02.921855 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.919790 5719 generic.go:356] "Generic (PLEG): container finished" podID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerID="3dd916b4c3a9f2419b58f99df80e51e497fc5d350bbc95a4d188f6aa96e4a592" exitCode=60 Oct 31 01:15:02.921855 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.919873 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" event={"ID":"0cd0b818-27c4-4a4c-b593-e8bbf5051650","Type":"ContainerDied","Data":"3dd916b4c3a9f2419b58f99df80e51e497fc5d350bbc95a4d188f6aa96e4a592"} Oct 31 01:15:02.921855 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.920136 5719 scope.go:122] "RemoveContainer" containerID="3dd916b4c3a9f2419b58f99df80e51e497fc5d350bbc95a4d188f6aa96e4a592" Oct 31 01:15:02.922036 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.922014 5719 generic.go:356] "Generic (PLEG): container finished" podID="f9798576-77d5-4c89-90ba-2c092992010b" containerID="39c3b2ab2cc2e342d3a20ea2fc770dcfa7b03ee0530d55fc34c2777844a67eb0" exitCode=60 Oct 31 01:15:02.922174 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.922052 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" event={"ID":"f9798576-77d5-4c89-90ba-2c092992010b","Type":"ContainerDied","Data":"39c3b2ab2cc2e342d3a20ea2fc770dcfa7b03ee0530d55fc34c2777844a67eb0"} Oct 31 01:15:02.922247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:02.922229 5719 scope.go:122] "RemoveContainer" containerID="39c3b2ab2cc2e342d3a20ea2fc770dcfa7b03ee0530d55fc34c2777844a67eb0" Oct 31 01:15:03.136443 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:03.136413 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:15:03.930075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.930049 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" event={"ID":"8b798301-db3c-4a7f-81f2-528d286dc5fb","Type":"ContainerStarted","Data":"15365bf51676f2f7d38f36b0b6177b69c6e896c1649fc0593159080c570f0e66"} Oct 31 01:15:03.932010 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.931988 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" event={"ID":"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2","Type":"ContainerStarted","Data":"fa9e5778551c9022017d1ae358dfe236cf90baec2d5f48ae24457a15cd292cb1"} Oct 31 01:15:03.933908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.933884 5719 generic.go:356] "Generic (PLEG): container finished" podID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerID="103abca65b5f69440e3084cdf4dec0f716af73a0577ac3acb0f0252769f91520" exitCode=60 Oct 31 01:15:03.934004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.933915 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" event={"ID":"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b","Type":"ContainerDied","Data":"103abca65b5f69440e3084cdf4dec0f716af73a0577ac3acb0f0252769f91520"} Oct 31 01:15:03.934004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.933960 5719 scope.go:122] "RemoveContainer" containerID="3c1e9337ba596b9893be936fabae82e021eb0e8ae7f60b82064a64d38e65e71d" Oct 31 01:15:03.934195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.934174 5719 scope.go:122] "RemoveContainer" containerID="103abca65b5f69440e3084cdf4dec0f716af73a0577ac3acb0f0252769f91520" Oct 31 01:15:03.934423 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:03.934398 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg_odh-ai-gateway-infra(659c95e5-6d46-4d75-a6c5-ba0e9be38f0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" Oct 31 01:15:03.936109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.936091 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" event={"ID":"e94b5461-808b-4892-9573-2d81fc28db7c","Type":"ContainerStarted","Data":"8d628733fd228e096202482e8cec2b4963c6058d5db44847440581fc9ba21e04"} Oct 31 01:15:03.938050 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.938032 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" event={"ID":"65e53823-39bb-4b96-8677-efd71615f28f","Type":"ContainerStarted","Data":"6279d53110bcf9a0e86284de1b665b2a931ec39685ef6f7bdfdc7c942542f8ee"} Oct 31 01:15:03.940105 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.940075 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" event={"ID":"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89","Type":"ContainerStarted","Data":"3dcefe10c15dfc5c3d94dd2461bfef8075bc805a697b002341a832e43107f8c5"} Oct 31 01:15:03.941999 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.941976 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" event={"ID":"0cd0b818-27c4-4a4c-b593-e8bbf5051650","Type":"ContainerStarted","Data":"21eaf8b81c4721cfefb7e4764b820f40ea832cee8620925a112dab29f54e1d10"} Oct 31 01:15:03.943906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.943883 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" event={"ID":"f9798576-77d5-4c89-90ba-2c092992010b","Type":"ContainerStarted","Data":"74547be355560070b378a3ac8fb450171333ecb756d6e28adecf5d2a10b2a531"} Oct 31 01:15:03.946065 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.946043 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" event={"ID":"e89e07dd-bed4-4675-adc8-ea689e709b98","Type":"ContainerStarted","Data":"c1cbea22fef1dce999762928315396ddb47061502b0913b8bdc2ff7a09fd255e"} Oct 31 01:15:03.947723 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:03.947687 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" event={"ID":"707c53af-0fef-46f9-a204-a315e0933fe6","Type":"ContainerStarted","Data":"eaca4214da36f7307a8ae31282fe8710a5ee03b07522518ddbfafef09fe181e9"} Oct 31 01:15:04.190036 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.189987 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" podStartSLOduration=4.18997152 podStartE2EDuration="4.18997152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.189766179 +0000 UTC m=+2785.725908481" watchObservedRunningTime="2026-10-31 01:15:04.18997152 +0000 UTC m=+2785.726113824" Oct 31 01:15:04.420173 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.420129 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" podStartSLOduration=4.420114931 podStartE2EDuration="4.420114931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.418968911 +0000 UTC m=+2785.955111211" watchObservedRunningTime="2026-10-31 01:15:04.420114931 +0000 UTC m=+2785.956257247" Oct 31 01:15:04.642478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.642420 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" podStartSLOduration=4.642407255 podStartE2EDuration="4.642407255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.640029476 +0000 UTC m=+2786.176171782" watchObservedRunningTime="2026-10-31 01:15:04.642407255 +0000 UTC m=+2786.178549559" Oct 31 01:15:04.958766 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.958735 5719 generic.go:356] "Generic (PLEG): container finished" podID="707c53af-0fef-46f9-a204-a315e0933fe6" containerID="eaca4214da36f7307a8ae31282fe8710a5ee03b07522518ddbfafef09fe181e9" exitCode=60 Oct 31 01:15:04.959171 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.958838 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" event={"ID":"707c53af-0fef-46f9-a204-a315e0933fe6","Type":"ContainerDied","Data":"eaca4214da36f7307a8ae31282fe8710a5ee03b07522518ddbfafef09fe181e9"} Oct 31 01:15:04.959171 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.958884 5719 scope.go:122] "RemoveContainer" containerID="d300d9d79c7008ec5d8b2dce92a8542fd1eda7ca8098bac60e72c15a44ae5ac7" Oct 31 01:15:04.959270 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.959250 5719 scope.go:122] "RemoveContainer" containerID="eaca4214da36f7307a8ae31282fe8710a5ee03b07522518ddbfafef09fe181e9" Oct 31 01:15:04.959572 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.959542 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h_odh-ai-gateway-infra(707c53af-0fef-46f9-a204-a315e0933fe6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" Oct 31 01:15:04.961079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.961057 5719 generic.go:356] "Generic (PLEG): container finished" podID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerID="15365bf51676f2f7d38f36b0b6177b69c6e896c1649fc0593159080c570f0e66" exitCode=60 Oct 31 01:15:04.961187 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.961082 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" event={"ID":"8b798301-db3c-4a7f-81f2-528d286dc5fb","Type":"ContainerDied","Data":"15365bf51676f2f7d38f36b0b6177b69c6e896c1649fc0593159080c570f0e66"} Oct 31 01:15:04.961346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.961330 5719 scope.go:122] "RemoveContainer" containerID="15365bf51676f2f7d38f36b0b6177b69c6e896c1649fc0593159080c570f0e66" Oct 31 01:15:04.961547 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.961522 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx_odh-ai-gateway-infra(8b798301-db3c-4a7f-81f2-528d286dc5fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" Oct 31 01:15:04.963297 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.963277 5719 generic.go:356] "Generic (PLEG): container finished" podID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerID="fa9e5778551c9022017d1ae358dfe236cf90baec2d5f48ae24457a15cd292cb1" exitCode=60 Oct 31 01:15:04.963422 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.963298 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" event={"ID":"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2","Type":"ContainerDied","Data":"fa9e5778551c9022017d1ae358dfe236cf90baec2d5f48ae24457a15cd292cb1"} Oct 31 01:15:04.963577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.963557 5719 scope.go:122] "RemoveContainer" containerID="fa9e5778551c9022017d1ae358dfe236cf90baec2d5f48ae24457a15cd292cb1" Oct 31 01:15:04.963823 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.963800 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2_odh-ai-gateway-infra(33c5bca9-c67f-4ab0-845b-bd4f4fad77a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" Oct 31 01:15:04.965661 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.965642 5719 scope.go:122] "RemoveContainer" containerID="103abca65b5f69440e3084cdf4dec0f716af73a0577ac3acb0f0252769f91520" Oct 31 01:15:04.965921 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.965898 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg_odh-ai-gateway-infra(659c95e5-6d46-4d75-a6c5-ba0e9be38f0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" Oct 31 01:15:04.967615 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.967597 5719 generic.go:356] "Generic (PLEG): container finished" podID="e94b5461-808b-4892-9573-2d81fc28db7c" containerID="8d628733fd228e096202482e8cec2b4963c6058d5db44847440581fc9ba21e04" exitCode=60 Oct 31 01:15:04.967738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.967718 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" event={"ID":"e94b5461-808b-4892-9573-2d81fc28db7c","Type":"ContainerDied","Data":"8d628733fd228e096202482e8cec2b4963c6058d5db44847440581fc9ba21e04"} Oct 31 01:15:04.967909 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.967891 5719 scope.go:122] "RemoveContainer" containerID="8d628733fd228e096202482e8cec2b4963c6058d5db44847440581fc9ba21e04" Oct 31 01:15:04.968118 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.968101 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6_odh-ai-gateway-infra(e94b5461-808b-4892-9573-2d81fc28db7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" Oct 31 01:15:04.969859 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.969836 5719 generic.go:356] "Generic (PLEG): container finished" podID="65e53823-39bb-4b96-8677-efd71615f28f" containerID="6279d53110bcf9a0e86284de1b665b2a931ec39685ef6f7bdfdc7c942542f8ee" exitCode=60 Oct 31 01:15:04.969962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.969916 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" event={"ID":"65e53823-39bb-4b96-8677-efd71615f28f","Type":"ContainerDied","Data":"6279d53110bcf9a0e86284de1b665b2a931ec39685ef6f7bdfdc7c942542f8ee"} Oct 31 01:15:04.970079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.970065 5719 scope.go:122] "RemoveContainer" containerID="6279d53110bcf9a0e86284de1b665b2a931ec39685ef6f7bdfdc7c942542f8ee" Oct 31 01:15:04.970291 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.970266 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657_odh-ai-gateway-infra(65e53823-39bb-4b96-8677-efd71615f28f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" podUID="65e53823-39bb-4b96-8677-efd71615f28f" Oct 31 01:15:04.972109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.972088 5719 generic.go:356] "Generic (PLEG): container finished" podID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerID="3dcefe10c15dfc5c3d94dd2461bfef8075bc805a697b002341a832e43107f8c5" exitCode=60 Oct 31 01:15:04.972221 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.972177 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" event={"ID":"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89","Type":"ContainerDied","Data":"3dcefe10c15dfc5c3d94dd2461bfef8075bc805a697b002341a832e43107f8c5"} Oct 31 01:15:04.972434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.972414 5719 scope.go:122] "RemoveContainer" containerID="3dcefe10c15dfc5c3d94dd2461bfef8075bc805a697b002341a832e43107f8c5" Oct 31 01:15:04.973099 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.973075 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd_odh-ai-gateway-infra(8017edc8-ca0b-4ac0-aff4-86d78ef3ea89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" Oct 31 01:15:04.974756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.974708 5719 scope.go:122] "RemoveContainer" containerID="71f6f5c05d6938c8fad076fa4e11322b61c432441a204506e9247d695208170d" Oct 31 01:15:04.974921 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.974901 5719 generic.go:356] "Generic (PLEG): container finished" podID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerID="21eaf8b81c4721cfefb7e4764b820f40ea832cee8620925a112dab29f54e1d10" exitCode=60 Oct 31 01:15:04.975009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.974929 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" event={"ID":"0cd0b818-27c4-4a4c-b593-e8bbf5051650","Type":"ContainerDied","Data":"21eaf8b81c4721cfefb7e4764b820f40ea832cee8620925a112dab29f54e1d10"} Oct 31 01:15:04.975177 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.975162 5719 scope.go:122] "RemoveContainer" containerID="21eaf8b81c4721cfefb7e4764b820f40ea832cee8620925a112dab29f54e1d10" Oct 31 01:15:04.975362 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.975338 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw_odh-ai-gateway-infra(0cd0b818-27c4-4a4c-b593-e8bbf5051650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" Oct 31 01:15:04.976910 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.976892 5719 generic.go:356] "Generic (PLEG): container finished" podID="f9798576-77d5-4c89-90ba-2c092992010b" containerID="74547be355560070b378a3ac8fb450171333ecb756d6e28adecf5d2a10b2a531" exitCode=60 Oct 31 01:15:04.976981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.976969 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" event={"ID":"f9798576-77d5-4c89-90ba-2c092992010b","Type":"ContainerDied","Data":"74547be355560070b378a3ac8fb450171333ecb756d6e28adecf5d2a10b2a531"} Oct 31 01:15:04.977124 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.977111 5719 scope.go:122] "RemoveContainer" containerID="74547be355560070b378a3ac8fb450171333ecb756d6e28adecf5d2a10b2a531" Oct 31 01:15:04.977331 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.977311 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72_odh-ai-gateway-infra(f9798576-77d5-4c89-90ba-2c092992010b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" podUID="f9798576-77d5-4c89-90ba-2c092992010b" Oct 31 01:15:04.978664 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.978644 5719 generic.go:356] "Generic (PLEG): container finished" podID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerID="c1cbea22fef1dce999762928315396ddb47061502b0913b8bdc2ff7a09fd255e" exitCode=60 Oct 31 01:15:04.978768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.978673 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" event={"ID":"e89e07dd-bed4-4675-adc8-ea689e709b98","Type":"ContainerDied","Data":"c1cbea22fef1dce999762928315396ddb47061502b0913b8bdc2ff7a09fd255e"} Oct 31 01:15:04.978911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.978897 5719 scope.go:122] "RemoveContainer" containerID="c1cbea22fef1dce999762928315396ddb47061502b0913b8bdc2ff7a09fd255e" Oct 31 01:15:04.979093 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:04.979075 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9_odh-ai-gateway-infra(e89e07dd-bed4-4675-adc8-ea689e709b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" Oct 31 01:15:04.987729 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:04.987710 5719 scope.go:122] "RemoveContainer" containerID="078380c6160a87541ce66a56337fdbc4f172c37c2855e61b8e58294d967a621b" Oct 31 01:15:05.000029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.000013 5719 scope.go:122] "RemoveContainer" containerID="37b3bdbf729815f42604abf922b24b167cacb5db03540a04100c6360a0cf6fca" Oct 31 01:15:05.012601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.012583 5719 scope.go:122] "RemoveContainer" containerID="55287e64a12d89d6b40a1699e7049747001ef5abd6debe7e4e455cb16a8c0960" Oct 31 01:15:05.029094 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.029073 5719 scope.go:122] "RemoveContainer" containerID="7d862098d9dec41b2cec68498bbc219548034aa7a1a5548279f9c363e1ec203a" Oct 31 01:15:05.041092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.041061 5719 scope.go:122] "RemoveContainer" containerID="3dd916b4c3a9f2419b58f99df80e51e497fc5d350bbc95a4d188f6aa96e4a592" Oct 31 01:15:05.053035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.053018 5719 scope.go:122] "RemoveContainer" containerID="39c3b2ab2cc2e342d3a20ea2fc770dcfa7b03ee0530d55fc34c2777844a67eb0" Oct 31 01:15:05.066203 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.066170 5719 scope.go:122] "RemoveContainer" containerID="e8beb80f9057ce78a850571904b90ea63090cd179c16307caa16a66d9c82ea68" Oct 31 01:15:05.386120 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.386093 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:15:05.562513 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562442 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-certs\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.562653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562538 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-token\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-token\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.562653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562570 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-data\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.562769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562677 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-envoy\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.562808 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562776 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4aeae8-c3e2-4111-9931-afbd88a847db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4aeae8-c3e2-4111-9931-afbd88a847db-istiod-ca-cert\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.562808 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562778 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-certs" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:05.562908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562842 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-kube-api-access-pp7p9\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-kube-api-access-pp7p9\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.562908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562854 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-data" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:05.562908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562868 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-credential-socket\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.563060 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.562982 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-socket\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.563060 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563023 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-podinfo\") pod \"fe4aeae8-c3e2-4111-9931-afbd88a847db\" (UID: \"fe4aeae8-c3e2-4111-9931-afbd88a847db\") " Oct 31 01:15:05.563176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563078 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-credential-socket" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:05.563265 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563243 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-socket" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:05.563310 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563172 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4aeae8-c3e2-4111-9931-afbd88a847db-istiod-ca-cert" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:05.563809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563783 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4aeae8-c3e2-4111-9931-afbd88a847db-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.564075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563814 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.564075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563831 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.564075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563846 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.564075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.563861 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.565019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.564985 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-token" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:05.565123 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.565104 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-envoy" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:15:05.565123 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.565109 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-kube-api-access-pp7p9" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "kube-api-access-pp7p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:05.565603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.565587 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-podinfo" pod "fe4aeae8-c3e2-4111-9931-afbd88a847db" (UID: "fe4aeae8-c3e2-4111-9931-afbd88a847db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:15:05.665546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.665519 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7p9\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-kube-api-access-pp7p9\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.665546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.665545 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.665734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.665554 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.665734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.665564 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4aeae8-c3e2-4111-9931-afbd88a847db-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:05.987724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.987676 5719 scope.go:122] "RemoveContainer" containerID="8d628733fd228e096202482e8cec2b4963c6058d5db44847440581fc9ba21e04" Oct 31 01:15:05.988185 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:05.987995 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6_odh-ai-gateway-infra(e94b5461-808b-4892-9573-2d81fc28db7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" Oct 31 01:15:05.989738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.989683 5719 scope.go:122] "RemoveContainer" containerID="6279d53110bcf9a0e86284de1b665b2a931ec39685ef6f7bdfdc7c942542f8ee" Oct 31 01:15:05.989950 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:05.989930 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657_odh-ai-gateway-infra(65e53823-39bb-4b96-8677-efd71615f28f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" podUID="65e53823-39bb-4b96-8677-efd71615f28f" Oct 31 01:15:05.991248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.991223 5719 scope.go:122] "RemoveContainer" containerID="3dcefe10c15dfc5c3d94dd2461bfef8075bc805a697b002341a832e43107f8c5" Oct 31 01:15:05.991434 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:05.991417 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd_odh-ai-gateway-infra(8017edc8-ca0b-4ac0-aff4-86d78ef3ea89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" Oct 31 01:15:05.992870 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.992851 5719 scope.go:122] "RemoveContainer" containerID="21eaf8b81c4721cfefb7e4764b820f40ea832cee8620925a112dab29f54e1d10" Oct 31 01:15:05.993071 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:05.993052 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw_odh-ai-gateway-infra(0cd0b818-27c4-4a4c-b593-e8bbf5051650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" Oct 31 01:15:05.994470 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.994453 5719 scope.go:122] "RemoveContainer" containerID="74547be355560070b378a3ac8fb450171333ecb756d6e28adecf5d2a10b2a531" Oct 31 01:15:05.994736 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:05.994679 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72_odh-ai-gateway-infra(f9798576-77d5-4c89-90ba-2c092992010b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" podUID="f9798576-77d5-4c89-90ba-2c092992010b" Oct 31 01:15:05.996115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.996099 5719 scope.go:122] "RemoveContainer" containerID="c1cbea22fef1dce999762928315396ddb47061502b0913b8bdc2ff7a09fd255e" Oct 31 01:15:05.996302 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:05.996284 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9_odh-ai-gateway-infra(e89e07dd-bed4-4675-adc8-ea689e709b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" Oct 31 01:15:05.997191 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.997170 5719 generic.go:356] "Generic (PLEG): container finished" podID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerID="6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e" exitCode=0 Oct 31 01:15:05.997268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.997211 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" Oct 31 01:15:05.997268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.997216 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" event={"ID":"fe4aeae8-c3e2-4111-9931-afbd88a847db","Type":"ContainerDied","Data":"6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e"} Oct 31 01:15:05.997360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.997287 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn" event={"ID":"fe4aeae8-c3e2-4111-9931-afbd88a847db","Type":"ContainerDied","Data":"6782b2da9885d1e383e928e64c39c48874c9ccf4c8af7c1e6e8b784445eb1069"} Oct 31 01:15:05.997360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.997316 5719 scope.go:122] "RemoveContainer" containerID="6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e" Oct 31 01:15:05.999056 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:05.999041 5719 scope.go:122] "RemoveContainer" containerID="eaca4214da36f7307a8ae31282fe8710a5ee03b07522518ddbfafef09fe181e9" Oct 31 01:15:05.999261 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:05.999242 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h_odh-ai-gateway-infra(707c53af-0fef-46f9-a204-a315e0933fe6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" Oct 31 01:15:06.000586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:06.000566 5719 scope.go:122] "RemoveContainer" containerID="15365bf51676f2f7d38f36b0b6177b69c6e896c1649fc0593159080c570f0e66" Oct 31 01:15:06.000785 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:06.000767 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx_odh-ai-gateway-infra(8b798301-db3c-4a7f-81f2-528d286dc5fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" Oct 31 01:15:06.002110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:06.002094 5719 scope.go:122] "RemoveContainer" containerID="fa9e5778551c9022017d1ae358dfe236cf90baec2d5f48ae24457a15cd292cb1" Oct 31 01:15:06.002274 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:06.002258 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2_odh-ai-gateway-infra(33c5bca9-c67f-4ab0-845b-bd4f4fad77a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" Oct 31 01:15:06.009170 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:06.009154 5719 scope.go:122] "RemoveContainer" containerID="6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e" Oct 31 01:15:06.009386 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:06.009371 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e\": container with ID starting with 6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e not found: ID does not exist" containerID="6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e" Oct 31 01:15:06.009433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:06.009392 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e"} err="failed to get container status \"6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e\": rpc error: code = NotFound desc = could not find container \"6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e\": container with ID starting with 6e7d638b5f3baed6df3123cdc123ad0a2e57630edce7622bdf8d5aafa327840e not found: ID does not exist" Oct 31 01:15:06.499638 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:06.499601 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn"] Oct 31 01:15:06.502313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:06.502288 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-zcnsn"] Oct 31 01:15:06.606707 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:06.606662 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" path="/var/lib/kubelet/pods/fe4aeae8-c3e2-4111-9931-afbd88a847db/volumes" Oct 31 01:15:14.595927 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:14.595897 5719 scope.go:122] "RemoveContainer" containerID="103abca65b5f69440e3084cdf4dec0f716af73a0577ac3acb0f0252769f91520" Oct 31 01:15:14.596582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:14.595992 5719 scope.go:122] "RemoveContainer" containerID="21eaf8b81c4721cfefb7e4764b820f40ea832cee8620925a112dab29f54e1d10" Oct 31 01:15:14.596582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:14.596141 5719 scope.go:122] "RemoveContainer" containerID="eaca4214da36f7307a8ae31282fe8710a5ee03b07522518ddbfafef09fe181e9" Oct 31 01:15:14.596736 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:14.596605 5719 scope.go:122] "RemoveContainer" containerID="3dcefe10c15dfc5c3d94dd2461bfef8075bc805a697b002341a832e43107f8c5" Oct 31 01:15:15.064122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:15.064091 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" event={"ID":"707c53af-0fef-46f9-a204-a315e0933fe6","Type":"ContainerStarted","Data":"6c82bef1e1de45580d54d399b7b0fca0eb3e934ac448f0a443e377aa0af53ec7"} Oct 31 01:15:15.066011 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:15.065989 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" event={"ID":"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b","Type":"ContainerStarted","Data":"b5c87c06bfbbe48652e32fd75b5455ee248d500518d4284866a2824634053beb"} Oct 31 01:15:15.067921 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:15.067897 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" event={"ID":"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89","Type":"ContainerStarted","Data":"3bd566c7a5a8d49caf95bef097561ce6af415ea1be5da64bc2fc09a85722ec7e"} Oct 31 01:15:15.069593 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:15.069573 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" event={"ID":"0cd0b818-27c4-4a4c-b593-e8bbf5051650","Type":"ContainerStarted","Data":"b063e8d14c8d23e71cb422ab131fb0219b519cdf2e973a32cd82e76c519e5c4a"} Oct 31 01:15:15.595135 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:15.595110 5719 scope.go:122] "RemoveContainer" containerID="c1cbea22fef1dce999762928315396ddb47061502b0913b8bdc2ff7a09fd255e" Oct 31 01:15:15.595283 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:15.595266 5719 scope.go:122] "RemoveContainer" containerID="fa9e5778551c9022017d1ae358dfe236cf90baec2d5f48ae24457a15cd292cb1" Oct 31 01:15:16.079045 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.079014 5719 generic.go:356] "Generic (PLEG): container finished" podID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerID="3bd566c7a5a8d49caf95bef097561ce6af415ea1be5da64bc2fc09a85722ec7e" exitCode=60 Oct 31 01:15:16.079481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.079096 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" event={"ID":"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89","Type":"ContainerDied","Data":"3bd566c7a5a8d49caf95bef097561ce6af415ea1be5da64bc2fc09a85722ec7e"} Oct 31 01:15:16.079481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.079174 5719 scope.go:122] "RemoveContainer" containerID="3dcefe10c15dfc5c3d94dd2461bfef8075bc805a697b002341a832e43107f8c5" Oct 31 01:15:16.079609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.079524 5719 scope.go:122] "RemoveContainer" containerID="3bd566c7a5a8d49caf95bef097561ce6af415ea1be5da64bc2fc09a85722ec7e" Oct 31 01:15:16.079868 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:16.079842 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd_odh-ai-gateway-infra(8017edc8-ca0b-4ac0-aff4-86d78ef3ea89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" Oct 31 01:15:16.081102 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.081078 5719 generic.go:356] "Generic (PLEG): container finished" podID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerID="b063e8d14c8d23e71cb422ab131fb0219b519cdf2e973a32cd82e76c519e5c4a" exitCode=60 Oct 31 01:15:16.081206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.081123 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" event={"ID":"0cd0b818-27c4-4a4c-b593-e8bbf5051650","Type":"ContainerDied","Data":"b063e8d14c8d23e71cb422ab131fb0219b519cdf2e973a32cd82e76c519e5c4a"} Oct 31 01:15:16.081421 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.081406 5719 scope.go:122] "RemoveContainer" containerID="b063e8d14c8d23e71cb422ab131fb0219b519cdf2e973a32cd82e76c519e5c4a" Oct 31 01:15:16.081671 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:16.081651 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw_odh-ai-gateway-infra(0cd0b818-27c4-4a4c-b593-e8bbf5051650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" Oct 31 01:15:16.083151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.083130 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" event={"ID":"e89e07dd-bed4-4675-adc8-ea689e709b98","Type":"ContainerStarted","Data":"657f7f46894201e16f497037707bb9028eff52db5ced542efa930035e482d298"} Oct 31 01:15:16.085027 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.085010 5719 generic.go:356] "Generic (PLEG): container finished" podID="707c53af-0fef-46f9-a204-a315e0933fe6" containerID="6c82bef1e1de45580d54d399b7b0fca0eb3e934ac448f0a443e377aa0af53ec7" exitCode=60 Oct 31 01:15:16.085111 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.085084 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" event={"ID":"707c53af-0fef-46f9-a204-a315e0933fe6","Type":"ContainerDied","Data":"6c82bef1e1de45580d54d399b7b0fca0eb3e934ac448f0a443e377aa0af53ec7"} Oct 31 01:15:16.085364 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.085348 5719 scope.go:122] "RemoveContainer" containerID="6c82bef1e1de45580d54d399b7b0fca0eb3e934ac448f0a443e377aa0af53ec7" Oct 31 01:15:16.085571 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:16.085546 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h_odh-ai-gateway-infra(707c53af-0fef-46f9-a204-a315e0933fe6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" Oct 31 01:15:16.087029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.087009 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" event={"ID":"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2","Type":"ContainerStarted","Data":"bb935b8db5b1d79ba061f4ab5eeba6085163fa034e547aee5aeb2b382f94630f"} Oct 31 01:15:16.088853 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.088835 5719 generic.go:356] "Generic (PLEG): container finished" podID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerID="b5c87c06bfbbe48652e32fd75b5455ee248d500518d4284866a2824634053beb" exitCode=60 Oct 31 01:15:16.088949 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.088911 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" event={"ID":"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b","Type":"ContainerDied","Data":"b5c87c06bfbbe48652e32fd75b5455ee248d500518d4284866a2824634053beb"} Oct 31 01:15:16.089193 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.089172 5719 scope.go:122] "RemoveContainer" containerID="b5c87c06bfbbe48652e32fd75b5455ee248d500518d4284866a2824634053beb" Oct 31 01:15:16.089392 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:16.089374 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg_odh-ai-gateway-infra(659c95e5-6d46-4d75-a6c5-ba0e9be38f0b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" Oct 31 01:15:16.092404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.092384 5719 scope.go:122] "RemoveContainer" containerID="21eaf8b81c4721cfefb7e4764b820f40ea832cee8620925a112dab29f54e1d10" Oct 31 01:15:16.106676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.106655 5719 scope.go:122] "RemoveContainer" containerID="eaca4214da36f7307a8ae31282fe8710a5ee03b07522518ddbfafef09fe181e9" Oct 31 01:15:16.120007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.119991 5719 scope.go:122] "RemoveContainer" containerID="103abca65b5f69440e3084cdf4dec0f716af73a0577ac3acb0f0252769f91520" Oct 31 01:15:16.195515 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.195485 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h"] Oct 31 01:15:16.301439 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.301375 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw"] Oct 31 01:15:16.431379 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.431348 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd"] Oct 31 01:15:16.538213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.538191 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg"] Oct 31 01:15:16.595395 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:16.595368 5719 scope.go:122] "RemoveContainer" containerID="6279d53110bcf9a0e86284de1b665b2a931ec39685ef6f7bdfdc7c942542f8ee" Oct 31 01:15:17.097969 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.097938 5719 generic.go:356] "Generic (PLEG): container finished" podID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerID="bb935b8db5b1d79ba061f4ab5eeba6085163fa034e547aee5aeb2b382f94630f" exitCode=60 Oct 31 01:15:17.098385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.098012 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" event={"ID":"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2","Type":"ContainerDied","Data":"bb935b8db5b1d79ba061f4ab5eeba6085163fa034e547aee5aeb2b382f94630f"} Oct 31 01:15:17.098385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.098068 5719 scope.go:122] "RemoveContainer" containerID="fa9e5778551c9022017d1ae358dfe236cf90baec2d5f48ae24457a15cd292cb1" Oct 31 01:15:17.098486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.098413 5719 scope.go:122] "RemoveContainer" containerID="bb935b8db5b1d79ba061f4ab5eeba6085163fa034e547aee5aeb2b382f94630f" Oct 31 01:15:17.098757 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:17.098724 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2_odh-ai-gateway-infra(33c5bca9-c67f-4ab0-845b-bd4f4fad77a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" Oct 31 01:15:17.102163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.102139 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" event={"ID":"65e53823-39bb-4b96-8677-efd71615f28f","Type":"ContainerStarted","Data":"a658208bd4403ee8de8ae75a2addcef578e3cd10a0e31edbcf0472df58ef0b93"} Oct 31 01:15:17.113200 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.113174 5719 generic.go:356] "Generic (PLEG): container finished" podID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerID="657f7f46894201e16f497037707bb9028eff52db5ced542efa930035e482d298" exitCode=60 Oct 31 01:15:17.113303 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.113203 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" event={"ID":"e89e07dd-bed4-4675-adc8-ea689e709b98","Type":"ContainerDied","Data":"657f7f46894201e16f497037707bb9028eff52db5ced542efa930035e482d298"} Oct 31 01:15:17.113672 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.113577 5719 scope.go:122] "RemoveContainer" containerID="657f7f46894201e16f497037707bb9028eff52db5ced542efa930035e482d298" Oct 31 01:15:17.113833 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.113811 5719 scope.go:122] "RemoveContainer" containerID="c1cbea22fef1dce999762928315396ddb47061502b0913b8bdc2ff7a09fd255e" Oct 31 01:15:17.114172 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:17.113880 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9_odh-ai-gateway-infra(e89e07dd-bed4-4675-adc8-ea689e709b98)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" Oct 31 01:15:17.293041 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.293011 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:17.299499 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.299473 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:17.393943 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.393920 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:17.396766 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.396746 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-kube-api-access-pwjsm\" (UniqueName: \"kubernetes.io/projected/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-kube-api-access-pwjsm\") pod \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\" (UID: \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\") " Oct 31 01:15:17.396892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.396874 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-maas-api-service-ca\") pod \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\" (UID: \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\") " Oct 31 01:15:17.397009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.396991 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-maas-api-service-ca\") pod \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\" (UID: \"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89\") " Oct 31 01:15:17.397133 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.397019 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-kube-api-access-lqf67\" (UniqueName: \"kubernetes.io/projected/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-kube-api-access-lqf67\") pod \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\" (UID: \"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b\") " Oct 31 01:15:17.397318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.397282 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-maas-api-service-ca" pod "659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" (UID: "659c95e5-6d46-4d75-a6c5-ba0e9be38f0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:17.397411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.397389 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-maas-api-service-ca" pod "8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" (UID: "8017edc8-ca0b-4ac0-aff4-86d78ef3ea89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:17.397544 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.397516 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:17.397748 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.397727 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.397837 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.397755 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.398996 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.398977 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-kube-api-access-pwjsm" pod "8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" (UID: "8017edc8-ca0b-4ac0-aff4-86d78ef3ea89"). InnerVolumeSpecName "kube-api-access-pwjsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:17.399200 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.399183 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-kube-api-access-lqf67" pod "659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" (UID: "659c95e5-6d46-4d75-a6c5-ba0e9be38f0b"). InnerVolumeSpecName "kube-api-access-lqf67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:17.499129 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499064 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cd0b818-27c4-4a4c-b593-e8bbf5051650-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0b818-27c4-4a4c-b593-e8bbf5051650-maas-api-service-ca\") pod \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\" (UID: \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\") " Oct 31 01:15:17.499129 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499116 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707c53af-0fef-46f9-a204-a315e0933fe6-kube-api-access-l2dll\" (UniqueName: \"kubernetes.io/projected/707c53af-0fef-46f9-a204-a315e0933fe6-kube-api-access-l2dll\") pod \"707c53af-0fef-46f9-a204-a315e0933fe6\" (UID: \"707c53af-0fef-46f9-a204-a315e0933fe6\") " Oct 31 01:15:17.499315 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499221 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707c53af-0fef-46f9-a204-a315e0933fe6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707c53af-0fef-46f9-a204-a315e0933fe6-maas-api-service-ca\") pod \"707c53af-0fef-46f9-a204-a315e0933fe6\" (UID: \"707c53af-0fef-46f9-a204-a315e0933fe6\") " Oct 31 01:15:17.499426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499408 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd0b818-27c4-4a4c-b593-e8bbf5051650-kube-api-access-5x7ch\" (UniqueName: \"kubernetes.io/projected/0cd0b818-27c4-4a4c-b593-e8bbf5051650-kube-api-access-5x7ch\") pod \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\" (UID: \"0cd0b818-27c4-4a4c-b593-e8bbf5051650\") " Oct 31 01:15:17.499513 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499492 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cd0b818-27c4-4a4c-b593-e8bbf5051650-maas-api-service-ca" pod "0cd0b818-27c4-4a4c-b593-e8bbf5051650" (UID: "0cd0b818-27c4-4a4c-b593-e8bbf5051650"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:17.499592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499572 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707c53af-0fef-46f9-a204-a315e0933fe6-maas-api-service-ca" pod "707c53af-0fef-46f9-a204-a315e0933fe6" (UID: "707c53af-0fef-46f9-a204-a315e0933fe6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:17.499908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499888 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707c53af-0fef-46f9-a204-a315e0933fe6-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.499908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499906 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjsm\" (UniqueName: \"kubernetes.io/projected/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89-kube-api-access-pwjsm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.500099 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499917 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqf67\" (UniqueName: \"kubernetes.io/projected/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b-kube-api-access-lqf67\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.500099 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.499925 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cd0b818-27c4-4a4c-b593-e8bbf5051650-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.501627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.501602 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707c53af-0fef-46f9-a204-a315e0933fe6-kube-api-access-l2dll" pod "707c53af-0fef-46f9-a204-a315e0933fe6" (UID: "707c53af-0fef-46f9-a204-a315e0933fe6"). InnerVolumeSpecName "kube-api-access-l2dll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:17.501758 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.501735 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd0b818-27c4-4a4c-b593-e8bbf5051650-kube-api-access-5x7ch" pod "0cd0b818-27c4-4a4c-b593-e8bbf5051650" (UID: "0cd0b818-27c4-4a4c-b593-e8bbf5051650"). InnerVolumeSpecName "kube-api-access-5x7ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:17.596754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.595143 5719 scope.go:122] "RemoveContainer" containerID="15365bf51676f2f7d38f36b0b6177b69c6e896c1649fc0593159080c570f0e66" Oct 31 01:15:17.596754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.595307 5719 scope.go:122] "RemoveContainer" containerID="74547be355560070b378a3ac8fb450171333ecb756d6e28adecf5d2a10b2a531" Oct 31 01:15:17.596754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.595431 5719 scope.go:122] "RemoveContainer" containerID="8d628733fd228e096202482e8cec2b4963c6058d5db44847440581fc9ba21e04" Oct 31 01:15:17.601565 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.601543 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x7ch\" (UniqueName: \"kubernetes.io/projected/0cd0b818-27c4-4a4c-b593-e8bbf5051650-kube-api-access-5x7ch\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.601618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.601572 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2dll\" (UniqueName: \"kubernetes.io/projected/707c53af-0fef-46f9-a204-a315e0933fe6-kube-api-access-l2dll\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.675928 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.675893 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2"] Oct 31 01:15:17.783961 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:17.783902 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9"] Oct 31 01:15:18.125747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.125718 5719 generic.go:356] "Generic (PLEG): container finished" podID="65e53823-39bb-4b96-8677-efd71615f28f" containerID="a658208bd4403ee8de8ae75a2addcef578e3cd10a0e31edbcf0472df58ef0b93" exitCode=60 Oct 31 01:15:18.126216 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.125786 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" event={"ID":"65e53823-39bb-4b96-8677-efd71615f28f","Type":"ContainerDied","Data":"a658208bd4403ee8de8ae75a2addcef578e3cd10a0e31edbcf0472df58ef0b93"} Oct 31 01:15:18.126216 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.125845 5719 scope.go:122] "RemoveContainer" containerID="6279d53110bcf9a0e86284de1b665b2a931ec39685ef6f7bdfdc7c942542f8ee" Oct 31 01:15:18.126216 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.126210 5719 scope.go:122] "RemoveContainer" containerID="a658208bd4403ee8de8ae75a2addcef578e3cd10a0e31edbcf0472df58ef0b93" Oct 31 01:15:18.126532 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:18.126509 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657_odh-ai-gateway-infra(65e53823-39bb-4b96-8677-efd71615f28f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" podUID="65e53823-39bb-4b96-8677-efd71615f28f" Oct 31 01:15:18.127405 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.127387 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" Oct 31 01:15:18.127781 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.127760 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd" event={"ID":"8017edc8-ca0b-4ac0-aff4-86d78ef3ea89","Type":"ContainerDied","Data":"fb88a0e96ff9b5b261369d47ec44310349ba1ec5d85d5bd43095b1f9f5b7822f"} Oct 31 01:15:18.129268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.129242 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" Oct 31 01:15:18.129385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.129284 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw" event={"ID":"0cd0b818-27c4-4a4c-b593-e8bbf5051650","Type":"ContainerDied","Data":"307ccf09c0cda41c9e0f29709a9eb2ec2344eada254274862e999c82008d4679"} Oct 31 01:15:18.131594 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.131573 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" event={"ID":"f9798576-77d5-4c89-90ba-2c092992010b","Type":"ContainerStarted","Data":"89606b9934f5d8d2e762218fca337d5f2ef6e51fc0e0772f486df5ff1c0c90bd"} Oct 31 01:15:18.135550 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.135524 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" event={"ID":"707c53af-0fef-46f9-a204-a315e0933fe6","Type":"ContainerDied","Data":"4933a40784bceb26604ed9fb423f971222c281e584b5a9709e3017c12ab8b0a6"} Oct 31 01:15:18.135639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.135614 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h" Oct 31 01:15:18.138807 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.138562 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" event={"ID":"8b798301-db3c-4a7f-81f2-528d286dc5fb","Type":"ContainerStarted","Data":"81f2471e1f0601add04fdc4111d74e10af1ad4f4e5f53aaa235b5a7907bd6d08"} Oct 31 01:15:18.142712 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.142673 5719 scope.go:122] "RemoveContainer" containerID="3bd566c7a5a8d49caf95bef097561ce6af415ea1be5da64bc2fc09a85722ec7e" Oct 31 01:15:18.147924 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.147899 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" event={"ID":"659c95e5-6d46-4d75-a6c5-ba0e9be38f0b","Type":"ContainerDied","Data":"eb7b87fdd1d8514bbadc404cf2489d1497042a8e09a02bd03a42b18ef05dcb7a"} Oct 31 01:15:18.148491 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.148473 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg" Oct 31 01:15:18.150528 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.150506 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" event={"ID":"e94b5461-808b-4892-9573-2d81fc28db7c","Type":"ContainerStarted","Data":"191a541f3043a703e4d7850af55b8c935feecf69b8ced367f076ae1d7df3e3dd"} Oct 31 01:15:18.160040 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.160007 5719 scope.go:122] "RemoveContainer" containerID="b063e8d14c8d23e71cb422ab131fb0219b519cdf2e973a32cd82e76c519e5c4a" Oct 31 01:15:18.172981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.172639 5719 scope.go:122] "RemoveContainer" containerID="6c82bef1e1de45580d54d399b7b0fca0eb3e934ac448f0a443e377aa0af53ec7" Oct 31 01:15:18.183014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.182955 5719 scope.go:122] "RemoveContainer" containerID="b5c87c06bfbbe48652e32fd75b5455ee248d500518d4284866a2824634053beb" Oct 31 01:15:18.320523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.320498 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:18.390015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.389951 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:18.415195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.415169 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89e07dd-bed4-4675-adc8-ea689e709b98-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e07dd-bed4-4675-adc8-ea689e709b98-maas-api-service-ca\") pod \"e89e07dd-bed4-4675-adc8-ea689e709b98\" (UID: \"e89e07dd-bed4-4675-adc8-ea689e709b98\") " Oct 31 01:15:18.415335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.415204 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89e07dd-bed4-4675-adc8-ea689e709b98-kube-api-access-sbqxt\" (UniqueName: \"kubernetes.io/projected/e89e07dd-bed4-4675-adc8-ea689e709b98-kube-api-access-sbqxt\") pod \"e89e07dd-bed4-4675-adc8-ea689e709b98\" (UID: \"e89e07dd-bed4-4675-adc8-ea689e709b98\") " Oct 31 01:15:18.415522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.415502 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89e07dd-bed4-4675-adc8-ea689e709b98-maas-api-service-ca" pod "e89e07dd-bed4-4675-adc8-ea689e709b98" (UID: "e89e07dd-bed4-4675-adc8-ea689e709b98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:18.415650 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.415638 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e07dd-bed4-4675-adc8-ea689e709b98-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.417402 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.417384 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89e07dd-bed4-4675-adc8-ea689e709b98-kube-api-access-sbqxt" pod "e89e07dd-bed4-4675-adc8-ea689e709b98" (UID: "e89e07dd-bed4-4675-adc8-ea689e709b98"). InnerVolumeSpecName "kube-api-access-sbqxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:18.436543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.436516 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw"] Oct 31 01:15:18.440885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.440863 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3be6fa-29890155-49dbw"] Oct 31 01:15:18.516421 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.516395 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-kube-api-access-v4tz2\" (UniqueName: \"kubernetes.io/projected/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-kube-api-access-v4tz2\") pod \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\" (UID: \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\") " Oct 31 01:15:18.516510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.516439 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-maas-api-service-ca\") pod \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\" (UID: \"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2\") " Oct 31 01:15:18.516770 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.516755 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqxt\" (UniqueName: \"kubernetes.io/projected/e89e07dd-bed4-4675-adc8-ea689e709b98-kube-api-access-sbqxt\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.516847 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.516770 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-maas-api-service-ca" pod "33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" (UID: "33c5bca9-c67f-4ab0-845b-bd4f4fad77a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:18.601502 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.600967 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-kube-api-access-v4tz2" pod "33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" (UID: "33c5bca9-c67f-4ab0-845b-bd4f4fad77a2"). InnerVolumeSpecName "kube-api-access-v4tz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:18.605074 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.605050 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" path="/var/lib/kubelet/pods/0cd0b818-27c4-4a4c-b593-e8bbf5051650/volumes" Oct 31 01:15:18.617523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.617500 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4tz2\" (UniqueName: \"kubernetes.io/projected/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-kube-api-access-v4tz2\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.617607 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.617527 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.656798 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.656731 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h"] Oct 31 01:15:18.661819 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.661800 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890155-pc22h"] Oct 31 01:15:18.765907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.765888 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd"] Oct 31 01:15:18.769454 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.769431 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b63c24-29890155-q5wcd"] Oct 31 01:15:18.779014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:18.778930 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657"] Oct 31 01:15:19.103177 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.103150 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg"] Oct 31 01:15:19.108945 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.108920 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-8fb7f0-29890155-sp5bg"] Oct 31 01:15:19.173522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.173492 5719 generic.go:356] "Generic (PLEG): container finished" podID="f9798576-77d5-4c89-90ba-2c092992010b" containerID="89606b9934f5d8d2e762218fca337d5f2ef6e51fc0e0772f486df5ff1c0c90bd" exitCode=60 Oct 31 01:15:19.173949 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.173592 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" event={"ID":"f9798576-77d5-4c89-90ba-2c092992010b","Type":"ContainerDied","Data":"89606b9934f5d8d2e762218fca337d5f2ef6e51fc0e0772f486df5ff1c0c90bd"} Oct 31 01:15:19.173949 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.173655 5719 scope.go:122] "RemoveContainer" containerID="74547be355560070b378a3ac8fb450171333ecb756d6e28adecf5d2a10b2a531" Oct 31 01:15:19.174092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.174073 5719 scope.go:122] "RemoveContainer" containerID="89606b9934f5d8d2e762218fca337d5f2ef6e51fc0e0772f486df5ff1c0c90bd" Oct 31 01:15:19.174408 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:19.174388 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72_odh-ai-gateway-infra(f9798576-77d5-4c89-90ba-2c092992010b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" podUID="f9798576-77d5-4c89-90ba-2c092992010b" Oct 31 01:15:19.175125 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.175092 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" event={"ID":"e89e07dd-bed4-4675-adc8-ea689e709b98","Type":"ContainerDied","Data":"858181c169219b5d078045e0a960be65ad2f7880b9310c49dbeaae188b84aaa7"} Oct 31 01:15:19.175125 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.175123 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9" Oct 31 01:15:19.178659 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.178636 5719 generic.go:356] "Generic (PLEG): container finished" podID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerID="81f2471e1f0601add04fdc4111d74e10af1ad4f4e5f53aaa235b5a7907bd6d08" exitCode=60 Oct 31 01:15:19.178805 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.178747 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" event={"ID":"8b798301-db3c-4a7f-81f2-528d286dc5fb","Type":"ContainerDied","Data":"81f2471e1f0601add04fdc4111d74e10af1ad4f4e5f53aaa235b5a7907bd6d08"} Oct 31 01:15:19.179211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.179187 5719 scope.go:122] "RemoveContainer" containerID="81f2471e1f0601add04fdc4111d74e10af1ad4f4e5f53aaa235b5a7907bd6d08" Oct 31 01:15:19.179498 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:19.179472 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx_odh-ai-gateway-infra(8b798301-db3c-4a7f-81f2-528d286dc5fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" Oct 31 01:15:19.180621 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.180600 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" event={"ID":"33c5bca9-c67f-4ab0-845b-bd4f4fad77a2","Type":"ContainerDied","Data":"b2ac684b4565ef68f13fab1e17bb343297c111b8aca5ae93a08118a774c62989"} Oct 31 01:15:19.180794 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.180686 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2" Oct 31 01:15:19.184611 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.184589 5719 generic.go:356] "Generic (PLEG): container finished" podID="e94b5461-808b-4892-9573-2d81fc28db7c" containerID="191a541f3043a703e4d7850af55b8c935feecf69b8ced367f076ae1d7df3e3dd" exitCode=60 Oct 31 01:15:19.184743 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.184718 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" event={"ID":"e94b5461-808b-4892-9573-2d81fc28db7c","Type":"ContainerDied","Data":"191a541f3043a703e4d7850af55b8c935feecf69b8ced367f076ae1d7df3e3dd"} Oct 31 01:15:19.185115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.185100 5719 scope.go:122] "RemoveContainer" containerID="191a541f3043a703e4d7850af55b8c935feecf69b8ced367f076ae1d7df3e3dd" Oct 31 01:15:19.185335 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:19.185314 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6_odh-ai-gateway-infra(e94b5461-808b-4892-9573-2d81fc28db7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" Oct 31 01:15:19.190025 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.190006 5719 scope.go:122] "RemoveContainer" containerID="657f7f46894201e16f497037707bb9028eff52db5ced542efa930035e482d298" Oct 31 01:15:19.199780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.199744 5719 scope.go:122] "RemoveContainer" containerID="15365bf51676f2f7d38f36b0b6177b69c6e896c1649fc0593159080c570f0e66" Oct 31 01:15:19.213069 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.213044 5719 scope.go:122] "RemoveContainer" containerID="bb935b8db5b1d79ba061f4ab5eeba6085163fa034e547aee5aeb2b382f94630f" Oct 31 01:15:19.228092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.228067 5719 scope.go:122] "RemoveContainer" containerID="8d628733fd228e096202482e8cec2b4963c6058d5db44847440581fc9ba21e04" Oct 31 01:15:19.339014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.338991 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:19.426862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.426790 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e53823-39bb-4b96-8677-efd71615f28f-kube-api-access-9kmvk\" (UniqueName: \"kubernetes.io/projected/65e53823-39bb-4b96-8677-efd71615f28f-kube-api-access-9kmvk\") pod \"65e53823-39bb-4b96-8677-efd71615f28f\" (UID: \"65e53823-39bb-4b96-8677-efd71615f28f\") " Oct 31 01:15:19.427005 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.426896 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e53823-39bb-4b96-8677-efd71615f28f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e53823-39bb-4b96-8677-efd71615f28f-maas-api-service-ca\") pod \"65e53823-39bb-4b96-8677-efd71615f28f\" (UID: \"65e53823-39bb-4b96-8677-efd71615f28f\") " Oct 31 01:15:19.427245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.427223 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e53823-39bb-4b96-8677-efd71615f28f-maas-api-service-ca" pod "65e53823-39bb-4b96-8677-efd71615f28f" (UID: "65e53823-39bb-4b96-8677-efd71615f28f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:19.429025 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.429002 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e53823-39bb-4b96-8677-efd71615f28f-kube-api-access-9kmvk" pod "65e53823-39bb-4b96-8677-efd71615f28f" (UID: "65e53823-39bb-4b96-8677-efd71615f28f"). InnerVolumeSpecName "kube-api-access-9kmvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:19.528196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.528164 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kmvk\" (UniqueName: \"kubernetes.io/projected/65e53823-39bb-4b96-8677-efd71615f28f-kube-api-access-9kmvk\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:19.528196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.528189 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65e53823-39bb-4b96-8677-efd71615f28f-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:19.566698 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.566667 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6"] Oct 31 01:15:19.949862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.949829 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2"] Oct 31 01:15:19.952603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:19.952581 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890155-rqvr2"] Oct 31 01:15:20.005528 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.005492 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx"] Oct 31 01:15:20.168965 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.168933 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9"] Oct 31 01:15:20.173019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.172994 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-dd98f0-29890155-vw5f9"] Oct 31 01:15:20.195195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.195171 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" Oct 31 01:15:20.195554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.195213 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657" event={"ID":"65e53823-39bb-4b96-8677-efd71615f28f","Type":"ContainerDied","Data":"81348314a8c55383e06cc43a259f0e30e1ea76cf9fd11c38a7031f73755322c2"} Oct 31 01:15:20.195554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.195279 5719 scope.go:122] "RemoveContainer" containerID="a658208bd4403ee8de8ae75a2addcef578e3cd10a0e31edbcf0472df58ef0b93" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.267890 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k"] Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268751 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268769 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268781 5719 cpu_manager.go:414] "containerMap: removing container" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268788 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268808 5719 cpu_manager.go:414] "containerMap: removing container" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268815 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268825 5719 cpu_manager.go:414] "containerMap: removing container" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268832 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268845 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268852 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268863 5719 cpu_manager.go:414] "containerMap: removing container" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268872 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268892 5719 cpu_manager.go:414] "containerMap: removing container" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268900 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268911 5719 cpu_manager.go:414] "containerMap: removing container" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268918 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268930 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268937 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268949 5719 cpu_manager.go:414] "containerMap: removing container" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268956 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268975 5719 cpu_manager.go:414] "containerMap: removing container" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268983 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.268995 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269001 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269017 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269024 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269044 5719 cpu_manager.go:414] "containerMap: removing container" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269051 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269087 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269094 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269233 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4aeae8-c3e2-4111-9931-afbd88a847db" containerName="istio-proxy" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269246 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269257 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269270 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269279 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269288 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269297 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269306 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269314 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269323 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269331 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269339 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269348 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269357 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269367 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269376 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269386 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269397 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:15:20.269598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.269405 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:15:20.280429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.280408 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.332590 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.332559 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4397fc\"/\"openshift-service-ca.crt\"" Oct 31 01:15:20.332791 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.332673 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4397fc\"/\"kube-root-ca.crt\"" Oct 31 01:15:20.332877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.332826 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4397fc\"/\"auth-test-model-4397fc-kserve-self-signed-certs\"" Oct 31 01:15:20.332993 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.332939 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4397fc\"/\"default-dockercfg-rf8nk\"" Oct 31 01:15:20.334322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.334296 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k"] Oct 31 01:15:20.337326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.336949 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-dshm\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.337326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.336995 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-home\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.337326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.337035 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tmp-dir\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.337326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.337098 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tls-certs\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.337326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.337140 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-model-cache\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.337326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.337206 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbh47\" (UniqueName: \"kubernetes.io/projected/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-kube-api-access-rbh47\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.342589 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.342563 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72"] Oct 31 01:15:20.370932 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.370903 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:20.422864 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.420448 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:20.439478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.438472 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94b5461-808b-4892-9573-2d81fc28db7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94b5461-808b-4892-9573-2d81fc28db7c-maas-api-service-ca\") pod \"e94b5461-808b-4892-9573-2d81fc28db7c\" (UID: \"e94b5461-808b-4892-9573-2d81fc28db7c\") " Oct 31 01:15:20.439478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.438809 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94b5461-808b-4892-9573-2d81fc28db7c-kube-api-access-hjn5b\" (UniqueName: \"kubernetes.io/projected/e94b5461-808b-4892-9573-2d81fc28db7c-kube-api-access-hjn5b\") pod \"e94b5461-808b-4892-9573-2d81fc28db7c\" (UID: \"e94b5461-808b-4892-9573-2d81fc28db7c\") " Oct 31 01:15:20.439478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.439366 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-dshm\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.439478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.439432 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-home\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.439908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.439495 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tmp-dir\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.439908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.439577 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tls-certs\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.439908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.439638 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-model-cache\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.439908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.439644 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94b5461-808b-4892-9573-2d81fc28db7c-maas-api-service-ca" pod "e94b5461-808b-4892-9573-2d81fc28db7c" (UID: "e94b5461-808b-4892-9573-2d81fc28db7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.439908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.439834 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbh47\" (UniqueName: \"kubernetes.io/projected/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-kube-api-access-rbh47\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.440179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.439949 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94b5461-808b-4892-9573-2d81fc28db7c-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.440179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.440053 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-model-cache\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.443524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.440794 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-home\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.443524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.441060 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tmp-dir\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.450653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.445706 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tls-certs\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.450653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.446134 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-dshm\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.452289 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.452261 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94b5461-808b-4892-9573-2d81fc28db7c-kube-api-access-hjn5b" pod "e94b5461-808b-4892-9573-2d81fc28db7c" (UID: "e94b5461-808b-4892-9573-2d81fc28db7c"). InnerVolumeSpecName "kube-api-access-hjn5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.498877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.498842 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbh47\" (UniqueName: \"kubernetes.io/projected/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-kube-api-access-rbh47\") pod \"auth-test-model-4397fc-kserve-645ffdfc65-r6v7k\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.524970 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.524946 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:20.534890 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.534847 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657"] Oct 31 01:15:20.537475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.537452 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ba34eb-29890155-g6657"] Oct 31 01:15:20.540283 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.540264 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b798301-db3c-4a7f-81f2-528d286dc5fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b798301-db3c-4a7f-81f2-528d286dc5fb-maas-api-service-ca\") pod \"8b798301-db3c-4a7f-81f2-528d286dc5fb\" (UID: \"8b798301-db3c-4a7f-81f2-528d286dc5fb\") " Oct 31 01:15:20.540357 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.540338 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b798301-db3c-4a7f-81f2-528d286dc5fb-kube-api-access-xbl6r\" (UniqueName: \"kubernetes.io/projected/8b798301-db3c-4a7f-81f2-528d286dc5fb-kube-api-access-xbl6r\") pod \"8b798301-db3c-4a7f-81f2-528d286dc5fb\" (UID: \"8b798301-db3c-4a7f-81f2-528d286dc5fb\") " Oct 31 01:15:20.540596 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.540575 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b798301-db3c-4a7f-81f2-528d286dc5fb-maas-api-service-ca" pod "8b798301-db3c-4a7f-81f2-528d286dc5fb" (UID: "8b798301-db3c-4a7f-81f2-528d286dc5fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.540860 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.540833 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b798301-db3c-4a7f-81f2-528d286dc5fb-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.540860 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.540857 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjn5b\" (UniqueName: \"kubernetes.io/projected/e94b5461-808b-4892-9573-2d81fc28db7c-kube-api-access-hjn5b\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.542457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.542438 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b798301-db3c-4a7f-81f2-528d286dc5fb-kube-api-access-xbl6r" pod "8b798301-db3c-4a7f-81f2-528d286dc5fb" (UID: "8b798301-db3c-4a7f-81f2-528d286dc5fb"). InnerVolumeSpecName "kube-api-access-xbl6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.606063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.606013 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:20.607599 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.607569 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" path="/var/lib/kubelet/pods/33c5bca9-c67f-4ab0-845b-bd4f4fad77a2/volumes" Oct 31 01:15:20.608299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.608275 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" path="/var/lib/kubelet/pods/659c95e5-6d46-4d75-a6c5-ba0e9be38f0b/volumes" Oct 31 01:15:20.608840 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.608821 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e53823-39bb-4b96-8677-efd71615f28f" path="/var/lib/kubelet/pods/65e53823-39bb-4b96-8677-efd71615f28f/volumes" Oct 31 01:15:20.609786 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.609751 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" path="/var/lib/kubelet/pods/707c53af-0fef-46f9-a204-a315e0933fe6/volumes" Oct 31 01:15:20.610291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.610262 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" path="/var/lib/kubelet/pods/8017edc8-ca0b-4ac0-aff4-86d78ef3ea89/volumes" Oct 31 01:15:20.610802 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.610782 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" path="/var/lib/kubelet/pods/e89e07dd-bed4-4675-adc8-ea689e709b98/volumes" Oct 31 01:15:20.641305 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.641280 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9798576-77d5-4c89-90ba-2c092992010b-kube-api-access-nnj62\" (UniqueName: \"kubernetes.io/projected/f9798576-77d5-4c89-90ba-2c092992010b-kube-api-access-nnj62\") pod \"f9798576-77d5-4c89-90ba-2c092992010b\" (UID: \"f9798576-77d5-4c89-90ba-2c092992010b\") " Oct 31 01:15:20.641447 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.641342 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9798576-77d5-4c89-90ba-2c092992010b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9798576-77d5-4c89-90ba-2c092992010b-maas-api-service-ca\") pod \"f9798576-77d5-4c89-90ba-2c092992010b\" (UID: \"f9798576-77d5-4c89-90ba-2c092992010b\") " Oct 31 01:15:20.641803 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.641778 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9798576-77d5-4c89-90ba-2c092992010b-maas-api-service-ca" pod "f9798576-77d5-4c89-90ba-2c092992010b" (UID: "f9798576-77d5-4c89-90ba-2c092992010b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.641918 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.641904 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbl6r\" (UniqueName: \"kubernetes.io/projected/8b798301-db3c-4a7f-81f2-528d286dc5fb-kube-api-access-xbl6r\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.641986 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.641920 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9798576-77d5-4c89-90ba-2c092992010b-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.644125 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.644090 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9798576-77d5-4c89-90ba-2c092992010b-kube-api-access-nnj62" pod "f9798576-77d5-4c89-90ba-2c092992010b" (UID: "f9798576-77d5-4c89-90ba-2c092992010b"). InnerVolumeSpecName "kube-api-access-nnj62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:20.744842 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.744766 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnj62\" (UniqueName: \"kubernetes.io/projected/f9798576-77d5-4c89-90ba-2c092992010b-kube-api-access-nnj62\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:15:20.998359 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:20.998285 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k"] Oct 31 01:15:21.004179 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:15:21.004152 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4ce34e8_a8d0_4c06_ae51_b523cae9a9f2.slice/crio-1b9b4f20814dd07122719a2f91a7f4cb56a4a36d63129b9f6b2f2bccb9e29d23 WatchSource:0}: Error finding container 1b9b4f20814dd07122719a2f91a7f4cb56a4a36d63129b9f6b2f2bccb9e29d23: Status 404 returned error can't find the container with id 1b9b4f20814dd07122719a2f91a7f4cb56a4a36d63129b9f6b2f2bccb9e29d23 Oct 31 01:15:21.177855 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.177816 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:15:21.208596 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.208565 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" Oct 31 01:15:21.208596 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.208581 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx" event={"ID":"8b798301-db3c-4a7f-81f2-528d286dc5fb","Type":"ContainerDied","Data":"12235a13e2f917c9df3ae0ee3b7717c02e1fae9fa0cd2c6e794fd8417536b4a5"} Oct 31 01:15:21.209132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.208645 5719 scope.go:122] "RemoveContainer" containerID="81f2471e1f0601add04fdc4111d74e10af1ad4f4e5f53aaa235b5a7907bd6d08" Oct 31 01:15:21.210469 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.210444 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" event={"ID":"e94b5461-808b-4892-9573-2d81fc28db7c","Type":"ContainerDied","Data":"00e861d349251fad6e836f255a3a383ecfb0fc7f7b0c2f80b5292fd6eec1230b"} Oct 31 01:15:21.210614 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.210453 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6" Oct 31 01:15:21.213616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.213593 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" event={"ID":"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2","Type":"ContainerStarted","Data":"d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23"} Oct 31 01:15:21.213747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.213629 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" event={"ID":"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2","Type":"ContainerStarted","Data":"1b9b4f20814dd07122719a2f91a7f4cb56a4a36d63129b9f6b2f2bccb9e29d23"} Oct 31 01:15:21.214978 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.214958 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" event={"ID":"f9798576-77d5-4c89-90ba-2c092992010b","Type":"ContainerDied","Data":"2ed7871f208a827c9c6dee650aad0cd92ffd73b321c7d699a674f77121b9edfc"} Oct 31 01:15:21.215066 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.214972 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72" Oct 31 01:15:21.221345 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.221318 5719 scope.go:122] "RemoveContainer" containerID="191a541f3043a703e4d7850af55b8c935feecf69b8ced367f076ae1d7df3e3dd" Oct 31 01:15:21.231769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.231750 5719 scope.go:122] "RemoveContainer" containerID="89606b9934f5d8d2e762218fca337d5f2ef6e51fc0e0772f486df5ff1c0c90bd" Oct 31 01:15:21.375107 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.375081 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72"] Oct 31 01:15:21.380882 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.380848 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ef2572-29890155-g7k72"] Oct 31 01:15:21.491525 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.491472 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" podStartSLOduration=1.491457954 podStartE2EDuration="1.491457954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:21.490539724 +0000 UTC m=+2803.026682023" watchObservedRunningTime="2026-10-31 01:15:21.491457954 +0000 UTC m=+2803.027600277" Oct 31 01:15:21.597833 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.597765 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6"] Oct 31 01:15:21.604877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.604841 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4397fc-29890155-cxwt6"] Oct 31 01:15:21.606219 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.606200 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:21.706453 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.706419 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx"] Oct 31 01:15:21.710727 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:21.710686 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-66e263-29890155-jpskx"] Oct 31 01:15:22.619781 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:22.619750 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" path="/var/lib/kubelet/pods/8b798301-db3c-4a7f-81f2-528d286dc5fb/volumes" Oct 31 01:15:22.620438 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:22.620417 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" path="/var/lib/kubelet/pods/e94b5461-808b-4892-9573-2d81fc28db7c/volumes" Oct 31 01:15:22.620875 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:22.620852 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9798576-77d5-4c89-90ba-2c092992010b" path="/var/lib/kubelet/pods/f9798576-77d5-4c89-90ba-2c092992010b/volumes" Oct 31 01:15:30.607411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:30.607382 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:30.620026 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:30.620006 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:31.299583 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:31.299551 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:15:33.009618 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:15:33.009585 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:15:41.710941 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:41.710912 5719 scope.go:122] "RemoveContainer" containerID="a67b30fed721697a5e1f113a52a64a3642c2d2bb7efa38c18b64932a4331f2a5" Oct 31 01:15:41.720682 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:41.720656 5719 scope.go:122] "RemoveContainer" containerID="14e306aea031070ceac7aa5423df5c6c22332127865ea00b483f2faa470911a8" Oct 31 01:15:41.729375 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:41.729357 5719 scope.go:122] "RemoveContainer" containerID="b684b39ed35ea81254778d995c221e4b9736cd9742ff9c9494503d8757abe281" Oct 31 01:15:41.737618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:41.737603 5719 scope.go:122] "RemoveContainer" containerID="80e476d80243c60964a655a8cbc615883341f6b3ebc17d5d327fa4ec60bfb7a8" Oct 31 01:15:41.745745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:15:41.745729 5719 scope.go:122] "RemoveContainer" containerID="b66707efd269bd491d53abbb169f7a2ce1ae61f5a3d3f066fac682125f827f28" Oct 31 01:16:03.010472 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:16:03.010438 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:16:10.513204 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:16:10.513166 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:16:33.011081 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:16:33.011042 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:17:01.179985 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:17:01.179953 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:17:03.010054 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:17:03.010015 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:17:33.010789 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:17:33.010750 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:17:57.016112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:17:57.016079 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:18:00.159878 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:00.159846 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft"] Oct 31 01:18:00.160341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:00.160061 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" containerID="cri-o://e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f" gracePeriod=30 Oct 31 01:18:03.009360 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:18:03.009328 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:18:05.415638 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.415615 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:18:05.466071 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466007 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-podinfo\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466088 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-socket\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466121 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-credential-socket\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466305 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466207 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-certs\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466305 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466237 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914a635c-961a-4ecf-bc37-e67f25add9d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914a635c-961a-4ecf-bc37-e67f25add9d8-istiod-ca-cert\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466305 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466274 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-data\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466469 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466381 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-token\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-token\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466469 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466384 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-socket" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:05.466469 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466435 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-kube-api-access-t642j\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-kube-api-access-t642j\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466472 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-envoy\") pod \"914a635c-961a-4ecf-bc37-e67f25add9d8\" (UID: \"914a635c-961a-4ecf-bc37-e67f25add9d8\") " Oct 31 01:18:05.466627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466484 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-certs" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:05.466627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466500 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-credential-socket" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:05.466627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466612 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-data" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:05.466844 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466658 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914a635c-961a-4ecf-bc37-e67f25add9d8-istiod-ca-cert" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:05.466948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466928 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:05.467007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466955 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:05.467007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466970 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:05.467007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466984 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914a635c-961a-4ecf-bc37-e67f25add9d8-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:05.467007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.466998 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:05.468291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.468263 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-podinfo" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:18:05.468596 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.468579 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-envoy" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:05.468803 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.468783 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-token" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:05.468883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.468819 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-kube-api-access-t642j" pod "914a635c-961a-4ecf-bc37-e67f25add9d8" (UID: "914a635c-961a-4ecf-bc37-e67f25add9d8"). InnerVolumeSpecName "kube-api-access-t642j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:05.568562 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.568531 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:05.568562 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.568560 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t642j\" (UniqueName: \"kubernetes.io/projected/914a635c-961a-4ecf-bc37-e67f25add9d8-kube-api-access-t642j\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:05.568745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.568574 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:05.568745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:05.568586 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914a635c-961a-4ecf-bc37-e67f25add9d8-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:06.245948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.245912 5719 generic.go:356] "Generic (PLEG): container finished" podID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerID="e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f" exitCode=0 Oct 31 01:18:06.246115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.245955 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" Oct 31 01:18:06.246115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.245965 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" event={"ID":"914a635c-961a-4ecf-bc37-e67f25add9d8","Type":"ContainerDied","Data":"e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f"} Oct 31 01:18:06.246115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.246013 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft" event={"ID":"914a635c-961a-4ecf-bc37-e67f25add9d8","Type":"ContainerDied","Data":"1c506da83a2077e3485a8f76e45d38af3fb80f56de1f121ae414cab60d637c84"} Oct 31 01:18:06.246115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.246036 5719 scope.go:122] "RemoveContainer" containerID="e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f" Oct 31 01:18:06.256886 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.256846 5719 scope.go:122] "RemoveContainer" containerID="e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f" Oct 31 01:18:06.257143 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:18:06.257123 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f\": container with ID starting with e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f not found: ID does not exist" containerID="e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f" Oct 31 01:18:06.257214 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.257151 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f"} err="failed to get container status \"e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f\": rpc error: code = NotFound desc = could not find container \"e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f\": container with ID starting with e25d0822594f7f8dc2f8f1be74ff7a2f46f4904599e0b1f06bdc231bf33a617f not found: ID does not exist" Oct 31 01:18:06.453536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.453503 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft"] Oct 31 01:18:06.501913 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.501849 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-n7xft"] Oct 31 01:18:06.604003 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:06.603973 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" path="/var/lib/kubelet/pods/914a635c-961a-4ecf-bc37-e67f25add9d8/volumes" Oct 31 01:18:07.444502 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:07.444470 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt"] Oct 31 01:18:07.444680 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:07.444650 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" containerID="cri-o://18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722" gracePeriod=30 Oct 31 01:18:12.245866 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.245826 5719 patch_prober.go:29] interesting pod/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:18:12.246268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.245887 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:18:12.721536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.721513 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:18:12.837715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.837667 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-envoy\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.837715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.837717 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-kube-api-access-zm589\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-kube-api-access-zm589\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.837896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.837741 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-certs\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.837896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.837775 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6edb592-7b44-4e7b-801c-90c632be47a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6edb592-7b44-4e7b-801c-90c632be47a4-istiod-ca-cert\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.837896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.837842 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-data\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.837896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.837866 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-token\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-token\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.838116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.837929 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-podinfo\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.838116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.837981 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-socket\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.838116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.838078 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-credential-socket\") pod \"d6edb592-7b44-4e7b-801c-90c632be47a4\" (UID: \"d6edb592-7b44-4e7b-801c-90c632be47a4\") " Oct 31 01:18:12.838116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.838101 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-data" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:12.838335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.838113 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-certs" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:12.838671 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.838640 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-credential-socket" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:12.838831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.838759 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-socket" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:12.838831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.838769 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6edb592-7b44-4e7b-801c-90c632be47a4-istiod-ca-cert" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:12.840633 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.840605 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-envoy" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:18:12.840780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.840642 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-kube-api-access-zm589" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "kube-api-access-zm589". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:12.841014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.840990 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:12.841098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.841020 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:12.841098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.841039 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:12.841098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.841053 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:12.841098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.841074 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm589\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-kube-api-access-zm589\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:12.841098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.841091 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6edb592-7b44-4e7b-801c-90c632be47a4-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:12.841365 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.841105 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6edb592-7b44-4e7b-801c-90c632be47a4-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:12.841365 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.841271 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-podinfo" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:18:12.850532 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.850503 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-token" pod "d6edb592-7b44-4e7b-801c-90c632be47a4" (UID: "d6edb592-7b44-4e7b-801c-90c632be47a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:12.942535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.942509 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:12.942535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:12.942534 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6edb592-7b44-4e7b-801c-90c632be47a4-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:13.288221 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.288142 5719 generic.go:356] "Generic (PLEG): container finished" podID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerID="18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722" exitCode=0 Oct 31 01:18:13.288221 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.288204 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" Oct 31 01:18:13.288770 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.288225 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" event={"ID":"d6edb592-7b44-4e7b-801c-90c632be47a4","Type":"ContainerDied","Data":"18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722"} Oct 31 01:18:13.288770 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.288308 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt" event={"ID":"d6edb592-7b44-4e7b-801c-90c632be47a4","Type":"ContainerDied","Data":"f2a8bb6c40458de2f7f7628c05c4fa3da7cc2661c74572cd4cb6ed73fa893987"} Oct 31 01:18:13.288770 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.288337 5719 scope.go:122] "RemoveContainer" containerID="18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722" Oct 31 01:18:13.299585 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.299557 5719 scope.go:122] "RemoveContainer" containerID="18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722" Oct 31 01:18:13.299908 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:18:13.299882 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722\": container with ID starting with 18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722 not found: ID does not exist" containerID="18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722" Oct 31 01:18:13.299990 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.299916 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722"} err="failed to get container status \"18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722\": rpc error: code = NotFound desc = could not find container \"18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722\": container with ID starting with 18f4d54c030b8c8bbe7931e981f2d70b96504f2672122fa4609f943035ffe722 not found: ID does not exist" Oct 31 01:18:13.446786 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.446762 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt"] Oct 31 01:18:13.451039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:13.451017 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-jzzgt"] Oct 31 01:18:14.605369 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:14.605340 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" path="/var/lib/kubelet/pods/d6edb592-7b44-4e7b-801c-90c632be47a4/volumes" Oct 31 01:18:33.009646 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:18:33.009597 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:18:45.112788 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.112752 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:18:45.925817 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.925783 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7"] Oct 31 01:18:45.926295 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926283 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:45.926341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926297 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:45.926341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926316 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:45.926341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926321 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:45.926341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926332 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:18:45.926341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926336 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd0b818-27c4-4a4c-b593-e8bbf5051650" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926349 5719 cpu_manager.go:414] "containerMap: removing container" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926353 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c5bca9-c67f-4ab0-845b-bd4f4fad77a2" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926361 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926365 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89e07dd-bed4-4675-adc8-ea689e709b98" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926377 5719 cpu_manager.go:414] "containerMap: removing container" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926382 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926390 5719 cpu_manager.go:414] "containerMap: removing container" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926394 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707c53af-0fef-46f9-a204-a315e0933fe6" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926401 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926405 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926411 5719 cpu_manager.go:414] "containerMap: removing container" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926415 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926423 5719 cpu_manager.go:414] "containerMap: removing container" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926428 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926444 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926448 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926455 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926459 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926469 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926473 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926479 5719 cpu_manager.go:414] "containerMap: removing container" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926483 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926490 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:45.926500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926493 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926573 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926579 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926584 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926592 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e53823-39bb-4b96-8677-efd71615f28f" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926600 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926605 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="659c95e5-6d46-4d75-a6c5-ba0e9be38f0b" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926613 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="914a635c-961a-4ecf-bc37-e67f25add9d8" containerName="istio-proxy" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926619 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6edb592-7b44-4e7b-801c-90c632be47a4" containerName="istio-proxy" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926624 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8017edc8-ca0b-4ac0-aff4-86d78ef3ea89" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926630 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:45.927373 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.926635 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:45.931287 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.931268 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:45.991288 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:45.991259 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8ddd2c\"/\"kube-root-ca.crt\"" Oct 31 01:18:46.032535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.032509 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-8ddd2c\"/\"openshift-service-ca.crt\"" Oct 31 01:18:46.032723 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.032563 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8ddd2c\"/\"explicit-model-8ddd2c-kserve-self-signed-certs\"" Oct 31 01:18:46.032723 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.032681 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-8ddd2c\"/\"default-dockercfg-97z6h\"" Oct 31 01:18:46.035244 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.035225 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7"] Oct 31 01:18:46.059226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.059196 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-dshm\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.059360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.059341 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-home\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.059411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.059368 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-model-cache\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.059411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.059391 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tls-certs\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.059499 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.059421 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8wtb\" (UniqueName: \"kubernetes.io/projected/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-kube-api-access-s8wtb\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.059626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.059608 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tmp-dir\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.160811 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.160777 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-home\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.160811 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.160809 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-model-cache\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.161363 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.160833 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tls-certs\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.161363 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.160862 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8wtb\" (UniqueName: \"kubernetes.io/projected/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-kube-api-access-s8wtb\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.161363 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.160956 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tmp-dir\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.161363 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.161010 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-dshm\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.161363 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.161266 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-home\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.161363 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.161323 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-model-cache\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.161363 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.161363 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tmp-dir\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.163350 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.163328 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-dshm\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.163516 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.163499 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tls-certs\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.233036 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.232979 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8wtb\" (UniqueName: \"kubernetes.io/projected/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-kube-api-access-s8wtb\") pod \"explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.242868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.242844 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:46.634617 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:46.634586 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7"] Oct 31 01:18:46.640961 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:18:46.640935 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podece72b26_b9f6_4823_a4fd_0b2f2121f52a.slice/crio-c0cdacaa1c1f70d70372f640a40c0a1b3f2344d999bb0ff8e603d1772934ff89 WatchSource:0}: Error finding container c0cdacaa1c1f70d70372f640a40c0a1b3f2344d999bb0ff8e603d1772934ff89: Status 404 returned error can't find the container with id c0cdacaa1c1f70d70372f640a40c0a1b3f2344d999bb0ff8e603d1772934ff89 Oct 31 01:18:47.514245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:47.514205 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" event={"ID":"ece72b26-b9f6-4823-a4fd-0b2f2121f52a","Type":"ContainerStarted","Data":"38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba"} Oct 31 01:18:47.514245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:47.514248 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" event={"ID":"ece72b26-b9f6-4823-a4fd-0b2f2121f52a","Type":"ContainerStarted","Data":"c0cdacaa1c1f70d70372f640a40c0a1b3f2344d999bb0ff8e603d1772934ff89"} Oct 31 01:18:47.629668 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:47.629618 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" podStartSLOduration=2.629605621 podStartE2EDuration="2.629605621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:47.628279675 +0000 UTC m=+3009.164421989" watchObservedRunningTime="2026-10-31 01:18:47.629605621 +0000 UTC m=+3009.165747941" Oct 31 01:18:48.243337 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:48.243308 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:52.393070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.393031 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd"] Oct 31 01:18:52.394410 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394383 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:52.394410 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394411 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:52.394601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394428 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:52.394601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394436 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:52.394601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394502 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:52.394601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394510 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:52.394911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394893 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94b5461-808b-4892-9573-2d81fc28db7c" containerName="cleanup" Oct 31 01:18:52.394966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394915 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9798576-77d5-4c89-90ba-2c092992010b" containerName="cleanup" Oct 31 01:18:52.394966 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.394934 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b798301-db3c-4a7f-81f2-528d286dc5fb" containerName="cleanup" Oct 31 01:18:52.399652 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.399623 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.451856 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.451827 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd"] Oct 31 01:18:52.496364 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.496332 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:18:52.496364 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.496349 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:18:52.496525 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.496433 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:18:52.497742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.497727 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:18:52.523779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.523759 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdxd\" (UniqueName: \"kubernetes.io/projected/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-kube-api-access-2hdxd\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.523906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.523892 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.523960 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.523930 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.625421 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.625401 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdxd\" (UniqueName: \"kubernetes.io/projected/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-kube-api-access-2hdxd\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.625534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.625491 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.625534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.625524 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.626153 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.626134 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.627902 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.627884 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.684946 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.684894 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdxd\" (UniqueName: \"kubernetes.io/projected/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-kube-api-access-2hdxd\") pod \"maas-api-revoke-sub-45404bb8-gt5hd\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:52.714798 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:52.714777 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:53.098625 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:53.098592 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd"] Oct 31 01:18:53.106098 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:18:53.106072 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0a0a3d_95fb_4450_be28_28ed4ee5c589.slice/crio-667a0e56ebdf7e661bd435b98bdbea8f516663e6c9780b26b147d83c3530bb76 WatchSource:0}: Error finding container 667a0e56ebdf7e661bd435b98bdbea8f516663e6c9780b26b147d83c3530bb76: Status 404 returned error can't find the container with id 667a0e56ebdf7e661bd435b98bdbea8f516663e6c9780b26b147d83c3530bb76 Oct 31 01:18:53.549375 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:53.549343 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" event={"ID":"8f0a0a3d-95fb-4450-be28-28ed4ee5c589","Type":"ContainerStarted","Data":"12b721dd56f4a3c47ff225bbe13e5737711f5b0e0d02de0ce48c30c96e4d6a2c"} Oct 31 01:18:53.549729 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:53.549402 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" event={"ID":"8f0a0a3d-95fb-4450-be28-28ed4ee5c589","Type":"ContainerStarted","Data":"667a0e56ebdf7e661bd435b98bdbea8f516663e6c9780b26b147d83c3530bb76"} Oct 31 01:18:53.663343 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:53.663267 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" podStartSLOduration=1.663254295 podStartE2EDuration="1.663254295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:53.661288247 +0000 UTC m=+3015.197430556" watchObservedRunningTime="2026-10-31 01:18:53.663254295 +0000 UTC m=+3015.199396624" Oct 31 01:18:54.556429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:54.556394 5719 generic.go:356] "Generic (PLEG): container finished" podID="8f0a0a3d-95fb-4450-be28-28ed4ee5c589" containerID="12b721dd56f4a3c47ff225bbe13e5737711f5b0e0d02de0ce48c30c96e4d6a2c" exitCode=0 Oct 31 01:18:54.556860 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:54.556477 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" event={"ID":"8f0a0a3d-95fb-4450-be28-28ed4ee5c589","Type":"ContainerDied","Data":"12b721dd56f4a3c47ff225bbe13e5737711f5b0e0d02de0ce48c30c96e4d6a2c"} Oct 31 01:18:55.725675 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.725652 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:55.859382 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.859352 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-xks-ca\") pod \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " Oct 31 01:18:55.859572 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.859451 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-openshift-ca\") pod \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " Oct 31 01:18:55.859648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.859585 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-kube-api-access-2hdxd\" (UniqueName: \"kubernetes.io/projected/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-kube-api-access-2hdxd\") pod \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\" (UID: \"8f0a0a3d-95fb-4450-be28-28ed4ee5c589\") " Oct 31 01:18:55.859873 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.859843 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-openshift-ca" pod "8f0a0a3d-95fb-4450-be28-28ed4ee5c589" (UID: "8f0a0a3d-95fb-4450-be28-28ed4ee5c589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:18:55.860090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.860074 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:55.861683 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.861662 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-xks-ca" pod "8f0a0a3d-95fb-4450-be28-28ed4ee5c589" (UID: "8f0a0a3d-95fb-4450-be28-28ed4ee5c589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:18:55.861912 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.861895 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-kube-api-access-2hdxd" pod "8f0a0a3d-95fb-4450-be28-28ed4ee5c589" (UID: "8f0a0a3d-95fb-4450-be28-28ed4ee5c589"). InnerVolumeSpecName "kube-api-access-2hdxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:18:55.961566 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.961531 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:55.961566 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:55.961568 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hdxd\" (UniqueName: \"kubernetes.io/projected/8f0a0a3d-95fb-4450-be28-28ed4ee5c589-kube-api-access-2hdxd\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:18:56.243140 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:56.243074 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:56.255457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:56.255437 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:18:56.569990 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:56.569964 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" Oct 31 01:18:56.570168 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:56.570000 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd" event={"ID":"8f0a0a3d-95fb-4450-be28-28ed4ee5c589","Type":"ContainerDied","Data":"667a0e56ebdf7e661bd435b98bdbea8f516663e6c9780b26b147d83c3530bb76"} Oct 31 01:18:56.570168 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:56.570048 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="667a0e56ebdf7e661bd435b98bdbea8f516663e6c9780b26b147d83c3530bb76" Oct 31 01:18:56.582433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:18:56.582412 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:19:03.009986 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:19:03.009948 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:19:03.127341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.127309 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6"] Oct 31 01:19:03.127968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.127949 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8f0a0a3d-95fb-4450-be28-28ed4ee5c589" containerName="revoke-keys" Oct 31 01:19:03.128030 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.127969 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0a0a3d-95fb-4450-be28-28ed4ee5c589" containerName="revoke-keys" Oct 31 01:19:03.128125 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.128115 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0a0a3d-95fb-4450-be28-28ed4ee5c589" containerName="revoke-keys" Oct 31 01:19:03.131824 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.131809 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.185895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.185866 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:03.234132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.234099 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:19:03.234291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.234139 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:19:03.234291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.234184 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:19:03.236561 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.236541 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6"] Oct 31 01:19:03.237772 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.237756 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.237842 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.237821 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.237970 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.237957 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgzh\" (UniqueName: \"kubernetes.io/projected/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-kube-api-access-9vgzh\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.338898 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.338875 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.339023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.338946 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.339102 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.339090 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgzh\" (UniqueName: \"kubernetes.io/projected/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-kube-api-access-9vgzh\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.339624 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.339604 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.341426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.341403 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.400665 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.400637 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgzh\" (UniqueName: \"kubernetes.io/projected/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-kube-api-access-9vgzh\") pod \"maas-api-revoke-sub-917a3e10-6rwc6\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.445662 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.445642 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:03.822780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:03.822750 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6"] Oct 31 01:19:03.828835 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:19:03.828808 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d51b923_7c8d_4da4_93b9_cb8d05c1827c.slice/crio-f6b21115d4e96da9b6ab601bdd596d8df579081abb257be9af39d541c903690e WatchSource:0}: Error finding container f6b21115d4e96da9b6ab601bdd596d8df579081abb257be9af39d541c903690e: Status 404 returned error can't find the container with id f6b21115d4e96da9b6ab601bdd596d8df579081abb257be9af39d541c903690e Oct 31 01:19:04.627793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:04.627752 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" event={"ID":"2d51b923-7c8d-4da4-93b9-cb8d05c1827c","Type":"ContainerStarted","Data":"ff5ad7c75ceb3a3fef88fbfa94c2d6023fd40706d786ad656b09473f053a0024"} Oct 31 01:19:04.628185 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:04.627810 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" event={"ID":"2d51b923-7c8d-4da4-93b9-cb8d05c1827c","Type":"ContainerStarted","Data":"f6b21115d4e96da9b6ab601bdd596d8df579081abb257be9af39d541c903690e"} Oct 31 01:19:04.745207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:04.745166 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" podStartSLOduration=1.745153076 podStartE2EDuration="1.745153076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:04.743115087 +0000 UTC m=+3026.279257393" watchObservedRunningTime="2026-10-31 01:19:04.745153076 +0000 UTC m=+3026.281295381" Oct 31 01:19:05.635867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:05.635836 5719 generic.go:356] "Generic (PLEG): container finished" podID="2d51b923-7c8d-4da4-93b9-cb8d05c1827c" containerID="ff5ad7c75ceb3a3fef88fbfa94c2d6023fd40706d786ad656b09473f053a0024" exitCode=0 Oct 31 01:19:05.636273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:05.635923 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" event={"ID":"2d51b923-7c8d-4da4-93b9-cb8d05c1827c","Type":"ContainerDied","Data":"ff5ad7c75ceb3a3fef88fbfa94c2d6023fd40706d786ad656b09473f053a0024"} Oct 31 01:19:06.791706 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.791672 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:06.875238 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.875213 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-kube-api-access-9vgzh\" (UniqueName: \"kubernetes.io/projected/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-kube-api-access-9vgzh\") pod \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " Oct 31 01:19:06.875388 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.875271 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-openshift-ca\") pod \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " Oct 31 01:19:06.875461 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.875448 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-xks-ca\") pod \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\" (UID: \"2d51b923-7c8d-4da4-93b9-cb8d05c1827c\") " Oct 31 01:19:06.875662 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.875635 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-openshift-ca" pod "2d51b923-7c8d-4da4-93b9-cb8d05c1827c" (UID: "2d51b923-7c8d-4da4-93b9-cb8d05c1827c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:06.875901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.875875 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:06.877590 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.877567 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-xks-ca" pod "2d51b923-7c8d-4da4-93b9-cb8d05c1827c" (UID: "2d51b923-7c8d-4da4-93b9-cb8d05c1827c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:06.877683 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.877636 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-kube-api-access-9vgzh" pod "2d51b923-7c8d-4da4-93b9-cb8d05c1827c" (UID: "2d51b923-7c8d-4da4-93b9-cb8d05c1827c"). InnerVolumeSpecName "kube-api-access-9vgzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:06.977141 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.977085 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:06.977141 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:06.977108 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgzh\" (UniqueName: \"kubernetes.io/projected/2d51b923-7c8d-4da4-93b9-cb8d05c1827c-kube-api-access-9vgzh\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:07.649700 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:07.649672 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" Oct 31 01:19:07.649845 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:07.649673 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6" event={"ID":"2d51b923-7c8d-4da4-93b9-cb8d05c1827c","Type":"ContainerDied","Data":"f6b21115d4e96da9b6ab601bdd596d8df579081abb257be9af39d541c903690e"} Oct 31 01:19:07.649845 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:07.649813 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b21115d4e96da9b6ab601bdd596d8df579081abb257be9af39d541c903690e" Oct 31 01:19:25.376806 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.376766 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf"] Oct 31 01:19:25.377599 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.377576 5719 cpu_manager.go:414] "containerMap: removing container" podUID="2d51b923-7c8d-4da4-93b9-cb8d05c1827c" containerName="revoke-keys" Oct 31 01:19:25.377742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.377599 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d51b923-7c8d-4da4-93b9-cb8d05c1827c" containerName="revoke-keys" Oct 31 01:19:25.377822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.377777 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d51b923-7c8d-4da4-93b9-cb8d05c1827c" containerName="revoke-keys" Oct 31 01:19:25.381526 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.381508 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.435785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.435756 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf"] Oct 31 01:19:25.471296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.471269 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjfcd\" (UniqueName: \"kubernetes.io/projected/a11027f7-8602-4ce0-9cb8-e778be4e99fe-kube-api-access-mjfcd\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.471418 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.471314 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.471473 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.471454 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.480459 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.480428 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:25.480459 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.480446 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:19:25.480459 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.480434 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:19:25.480641 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.480469 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:19:25.572459 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.572436 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.572587 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.572538 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjfcd\" (UniqueName: \"kubernetes.io/projected/a11027f7-8602-4ce0-9cb8-e778be4e99fe-kube-api-access-mjfcd\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.572587 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.572578 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.573241 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.573224 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.574975 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.574954 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.631538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.631478 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjfcd\" (UniqueName: \"kubernetes.io/projected/a11027f7-8602-4ce0-9cb8-e778be4e99fe-kube-api-access-mjfcd\") pod \"maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:25.694110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:25.694094 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:26.071994 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:26.071963 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf"] Oct 31 01:19:26.077900 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:19:26.077868 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11027f7_8602_4ce0_9cb8_e778be4e99fe.slice/crio-64acad93992f6cc4129e10827f416515c1845213db54ee073a98208f9facc67d WatchSource:0}: Error finding container 64acad93992f6cc4129e10827f416515c1845213db54ee073a98208f9facc67d: Status 404 returned error can't find the container with id 64acad93992f6cc4129e10827f416515c1845213db54ee073a98208f9facc67d Oct 31 01:19:26.767342 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:26.767270 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" event={"ID":"a11027f7-8602-4ce0-9cb8-e778be4e99fe","Type":"ContainerStarted","Data":"a60e4b5eeadca7a22553a722fd5511bc4e4cabec4c4bf04f3aaa77788ed0d971"} Oct 31 01:19:26.767342 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:26.767330 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" event={"ID":"a11027f7-8602-4ce0-9cb8-e778be4e99fe","Type":"ContainerStarted","Data":"64acad93992f6cc4129e10827f416515c1845213db54ee073a98208f9facc67d"} Oct 31 01:19:26.878935 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:26.878886 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" podStartSLOduration=1.878872258 podStartE2EDuration="1.878872258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:26.878458524 +0000 UTC m=+3048.414600827" watchObservedRunningTime="2026-10-31 01:19:26.878872258 +0000 UTC m=+3048.415014562" Oct 31 01:19:27.775497 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:27.775468 5719 generic.go:356] "Generic (PLEG): container finished" podID="a11027f7-8602-4ce0-9cb8-e778be4e99fe" containerID="a60e4b5eeadca7a22553a722fd5511bc4e4cabec4c4bf04f3aaa77788ed0d971" exitCode=0 Oct 31 01:19:27.775904 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:27.775504 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" event={"ID":"a11027f7-8602-4ce0-9cb8-e778be4e99fe","Type":"ContainerDied","Data":"a60e4b5eeadca7a22553a722fd5511bc4e4cabec4c4bf04f3aaa77788ed0d971"} Oct 31 01:19:28.928469 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:28.928441 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:29.006073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.006050 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-openshift-ca\") pod \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " Oct 31 01:19:29.006205 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.006136 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11027f7-8602-4ce0-9cb8-e778be4e99fe-kube-api-access-mjfcd\" (UniqueName: \"kubernetes.io/projected/a11027f7-8602-4ce0-9cb8-e778be4e99fe-kube-api-access-mjfcd\") pod \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " Oct 31 01:19:29.006205 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.006196 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-xks-ca\") pod \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\" (UID: \"a11027f7-8602-4ce0-9cb8-e778be4e99fe\") " Oct 31 01:19:29.006412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.006391 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-openshift-ca" pod "a11027f7-8602-4ce0-9cb8-e778be4e99fe" (UID: "a11027f7-8602-4ce0-9cb8-e778be4e99fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:29.006727 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.006680 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:29.008227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.008200 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11027f7-8602-4ce0-9cb8-e778be4e99fe-kube-api-access-mjfcd" pod "a11027f7-8602-4ce0-9cb8-e778be4e99fe" (UID: "a11027f7-8602-4ce0-9cb8-e778be4e99fe"). InnerVolumeSpecName "kube-api-access-mjfcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:29.008337 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.008282 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-xks-ca" pod "a11027f7-8602-4ce0-9cb8-e778be4e99fe" (UID: "a11027f7-8602-4ce0-9cb8-e778be4e99fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:29.107803 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.107777 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjfcd\" (UniqueName: \"kubernetes.io/projected/a11027f7-8602-4ce0-9cb8-e778be4e99fe-kube-api-access-mjfcd\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:29.107803 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.107801 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11027f7-8602-4ce0-9cb8-e778be4e99fe-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:29.243584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.243555 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc"] Oct 31 01:19:29.789007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.788976 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" Oct 31 01:19:29.789176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.788975 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf" event={"ID":"a11027f7-8602-4ce0-9cb8-e778be4e99fe","Type":"ContainerDied","Data":"64acad93992f6cc4129e10827f416515c1845213db54ee073a98208f9facc67d"} Oct 31 01:19:29.789176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:29.789100 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64acad93992f6cc4129e10827f416515c1845213db54ee073a98208f9facc67d" Oct 31 01:19:33.010679 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:19:33.010645 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:19:41.866632 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:41.866601 5719 scope.go:122] "RemoveContainer" containerID="223d1eef5e2155d049b8c1f4f0c4213b43d7defc3f27df577825cb679abb60b1" Oct 31 01:19:44.265441 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.265405 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" podUID="a68090ac-329e-4b98-a6b4-77fb3cd7095a" containerName="main" containerID="cri-o://f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648" gracePeriod=45 Oct 31 01:19:44.401756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.401723 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" podUID="a68090ac-329e-4b98-a6b4-77fb3cd7095a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.183:8000/ready\": dial tcp 10.133.0.183:8000: connect: connection refused" Oct 31 01:19:44.560583 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.560558 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:19:44.667492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.667456 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tls-certs\") pod \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " Oct 31 01:19:44.667713 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.667515 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-dshm\") pod \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " Oct 31 01:19:44.667713 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.667547 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68090ac-329e-4b98-a6b4-77fb3cd7095a-kube-api-access-27flj\" (UniqueName: \"kubernetes.io/projected/a68090ac-329e-4b98-a6b4-77fb3cd7095a-kube-api-access-27flj\") pod \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " Oct 31 01:19:44.667713 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.667624 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-home\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-home\") pod \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " Oct 31 01:19:44.667713 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.667651 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tmp-dir\") pod \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " Oct 31 01:19:44.667954 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.667717 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-model-cache\") pod \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\" (UID: \"a68090ac-329e-4b98-a6b4-77fb3cd7095a\") " Oct 31 01:19:44.669281 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.669250 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:19:44.670315 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.670255 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-model-cache" pod "a68090ac-329e-4b98-a6b4-77fb3cd7095a" (UID: "a68090ac-329e-4b98-a6b4-77fb3cd7095a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:44.670471 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.670362 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tmp-dir" pod "a68090ac-329e-4b98-a6b4-77fb3cd7095a" (UID: "a68090ac-329e-4b98-a6b4-77fb3cd7095a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:44.670600 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.670546 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-home" pod "a68090ac-329e-4b98-a6b4-77fb3cd7095a" (UID: "a68090ac-329e-4b98-a6b4-77fb3cd7095a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:44.671482 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.671446 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tls-certs" pod "a68090ac-329e-4b98-a6b4-77fb3cd7095a" (UID: "a68090ac-329e-4b98-a6b4-77fb3cd7095a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:44.671656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.671631 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-dshm" pod "a68090ac-329e-4b98-a6b4-77fb3cd7095a" (UID: "a68090ac-329e-4b98-a6b4-77fb3cd7095a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:19:44.672587 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.672562 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68090ac-329e-4b98-a6b4-77fb3cd7095a-kube-api-access-27flj" pod "a68090ac-329e-4b98-a6b4-77fb3cd7095a" (UID: "a68090ac-329e-4b98-a6b4-77fb3cd7095a"). InnerVolumeSpecName "kube-api-access-27flj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:44.769519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.769488 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:44.769519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.769521 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:44.769802 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.769530 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27flj\" (UniqueName: \"kubernetes.io/projected/a68090ac-329e-4b98-a6b4-77fb3cd7095a-kube-api-access-27flj\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:44.769802 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.769541 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:44.769802 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.769551 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:44.769802 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.769558 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a68090ac-329e-4b98-a6b4-77fb3cd7095a-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:19:44.878249 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.878217 5719 generic.go:356] "Generic (PLEG): container finished" podID="a68090ac-329e-4b98-a6b4-77fb3cd7095a" containerID="f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648" exitCode=0 Oct 31 01:19:44.878446 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.878263 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" Oct 31 01:19:44.878446 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.878316 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" event={"ID":"a68090ac-329e-4b98-a6b4-77fb3cd7095a","Type":"ContainerDied","Data":"f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648"} Oct 31 01:19:44.878446 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.878356 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc" event={"ID":"a68090ac-329e-4b98-a6b4-77fb3cd7095a","Type":"ContainerDied","Data":"64ab3339eea85d904da8fa48633a715c67c75ae74ace398ca3679c9c4bced8d3"} Oct 31 01:19:44.878446 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.878376 5719 scope.go:122] "RemoveContainer" containerID="f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648" Oct 31 01:19:44.890772 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.890748 5719 scope.go:122] "RemoveContainer" containerID="f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648" Oct 31 01:19:44.891080 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:19:44.891056 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648\": container with ID starting with f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648 not found: ID does not exist" containerID="f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648" Oct 31 01:19:44.891163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:44.891090 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648"} err="failed to get container status \"f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648\": rpc error: code = NotFound desc = could not find container \"f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648\": container with ID starting with f1105f355f93a7b6216faf333b5640c4db5d6abbafb0c1164bd47625c6e20648 not found: ID does not exist" Oct 31 01:19:45.037715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:45.037664 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc"] Oct 31 01:19:45.044841 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:45.044817 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8fb7f0/auth-test-model-8fb7f0-kserve-868495f8d5-s2drc"] Oct 31 01:19:46.604046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:46.603971 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a68090ac-329e-4b98-a6b4-77fb3cd7095a" path="/var/lib/kubelet/pods/a68090ac-329e-4b98-a6b4-77fb3cd7095a/volumes" Oct 31 01:19:46.648719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:46.648673 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf"] Oct 31 01:19:46.667081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:46.667054 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-8ddd2c-w2cvf"] Oct 31 01:19:48.603704 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:48.603659 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11027f7-8602-4ce0-9cb8-e778be4e99fe" path="/var/lib/kubelet/pods/a11027f7-8602-4ce0-9cb8-e778be4e99fe/volumes" Oct 31 01:19:52.094840 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:19:52.094809 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7"] Oct 31 01:20:03.010448 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:20:03.010417 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:20:07.116105 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.116061 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" podUID="ece72b26-b9f6-4823-a4fd-0b2f2121f52a" containerName="main" containerID="cri-o://38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba" gracePeriod=45 Oct 31 01:20:07.379661 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.379603 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:20:07.495609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.495576 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-kube-api-access-s8wtb\" (UniqueName: \"kubernetes.io/projected/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-kube-api-access-s8wtb\") pod \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " Oct 31 01:20:07.495768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.495687 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tmp-dir\") pod \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " Oct 31 01:20:07.495768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.495732 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-model-cache\") pod \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " Oct 31 01:20:07.495768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.495754 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tls-certs\") pod \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " Oct 31 01:20:07.495947 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.495780 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-home\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-home\") pod \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " Oct 31 01:20:07.495947 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.495818 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-dshm\") pod \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\" (UID: \"ece72b26-b9f6-4823-a4fd-0b2f2121f52a\") " Oct 31 01:20:07.496646 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.496584 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-model-cache" pod "ece72b26-b9f6-4823-a4fd-0b2f2121f52a" (UID: "ece72b26-b9f6-4823-a4fd-0b2f2121f52a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:07.496849 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.496721 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tmp-dir" pod "ece72b26-b9f6-4823-a4fd-0b2f2121f52a" (UID: "ece72b26-b9f6-4823-a4fd-0b2f2121f52a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:07.497059 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.497039 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-home" pod "ece72b26-b9f6-4823-a4fd-0b2f2121f52a" (UID: "ece72b26-b9f6-4823-a4fd-0b2f2121f52a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:07.498708 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.498654 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-dshm" pod "ece72b26-b9f6-4823-a4fd-0b2f2121f52a" (UID: "ece72b26-b9f6-4823-a4fd-0b2f2121f52a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:07.499007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.498986 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-kube-api-access-s8wtb" pod "ece72b26-b9f6-4823-a4fd-0b2f2121f52a" (UID: "ece72b26-b9f6-4823-a4fd-0b2f2121f52a"). InnerVolumeSpecName "kube-api-access-s8wtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:07.499108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.499091 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tls-certs" pod "ece72b26-b9f6-4823-a4fd-0b2f2121f52a" (UID: "ece72b26-b9f6-4823-a4fd-0b2f2121f52a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:07.597676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.597647 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:07.597676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.597669 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:07.597676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.597679 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:07.597874 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.597705 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:07.597874 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.597715 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:07.597874 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:07.597725 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8wtb\" (UniqueName: \"kubernetes.io/projected/ece72b26-b9f6-4823-a4fd-0b2f2121f52a-kube-api-access-s8wtb\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:08.012777 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.012747 5719 generic.go:356] "Generic (PLEG): container finished" podID="ece72b26-b9f6-4823-a4fd-0b2f2121f52a" containerID="38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba" exitCode=0 Oct 31 01:20:08.012989 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.012796 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" Oct 31 01:20:08.012989 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.012839 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" event={"ID":"ece72b26-b9f6-4823-a4fd-0b2f2121f52a","Type":"ContainerDied","Data":"38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba"} Oct 31 01:20:08.012989 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.012885 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7" event={"ID":"ece72b26-b9f6-4823-a4fd-0b2f2121f52a","Type":"ContainerDied","Data":"c0cdacaa1c1f70d70372f640a40c0a1b3f2344d999bb0ff8e603d1772934ff89"} Oct 31 01:20:08.012989 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.012908 5719 scope.go:122] "RemoveContainer" containerID="38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba" Oct 31 01:20:08.023905 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.023889 5719 scope.go:122] "RemoveContainer" containerID="38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba" Oct 31 01:20:08.024177 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:20:08.024158 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba\": container with ID starting with 38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba not found: ID does not exist" containerID="38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba" Oct 31 01:20:08.024245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.024189 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba"} err="failed to get container status \"38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba\": rpc error: code = NotFound desc = could not find container \"38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba\": container with ID starting with 38dd0b32fab9cd356f523dd53a454a99e2a18d2cd4450335af3aef1cc06665ba not found: ID does not exist" Oct 31 01:20:08.170419 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.170392 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7"] Oct 31 01:20:08.177224 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.177200 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-8ddd2c/explicit-model-8ddd2c-kserve-6b9f5759d5-xfnf7"] Oct 31 01:20:08.603023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.602994 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece72b26-b9f6-4823-a4fd-0b2f2121f52a" path="/var/lib/kubelet/pods/ece72b26-b9f6-4823-a4fd-0b2f2121f52a/volumes" Oct 31 01:20:08.982360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:08.982273 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9"] Oct 31 01:20:15.361058 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361026 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7292-ff3a7e"] Oct 31 01:20:15.361560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361546 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a11027f7-8602-4ce0-9cb8-e778be4e99fe" containerName="revoke-keys" Oct 31 01:20:15.361606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361561 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11027f7-8602-4ce0-9cb8-e778be4e99fe" containerName="revoke-keys" Oct 31 01:20:15.361606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361583 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ece72b26-b9f6-4823-a4fd-0b2f2121f52a" containerName="main" Oct 31 01:20:15.361606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361588 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece72b26-b9f6-4823-a4fd-0b2f2121f52a" containerName="main" Oct 31 01:20:15.361606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361602 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a68090ac-329e-4b98-a6b4-77fb3cd7095a" containerName="main" Oct 31 01:20:15.361606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361606 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68090ac-329e-4b98-a6b4-77fb3cd7095a" containerName="main" Oct 31 01:20:15.361804 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361685 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11027f7-8602-4ce0-9cb8-e778be4e99fe" containerName="revoke-keys" Oct 31 01:20:15.361804 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361711 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece72b26-b9f6-4823-a4fd-0b2f2121f52a" containerName="main" Oct 31 01:20:15.361804 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.361718 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68090ac-329e-4b98-a6b4-77fb3cd7095a" containerName="main" Oct 31 01:20:15.366411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.366393 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-ff3a7e" Oct 31 01:20:15.382229 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.382210 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rmfm\" (UniqueName: \"kubernetes.io/projected/8d720a4f-4da7-4bd1-902f-c355ffd4fa70-kube-api-access-6rmfm\") pod \"test-curl-7292-ff3a7e\" (UID: \"8d720a4f-4da7-4bd1-902f-c355ffd4fa70\") " pod="opendatahub/test-curl-7292-ff3a7e" Oct 31 01:20:15.423639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.423618 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-ff3a7e"] Oct 31 01:20:15.445509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.445485 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8gxx2\"" Oct 31 01:20:15.482755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.482728 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rmfm\" (UniqueName: \"kubernetes.io/projected/8d720a4f-4da7-4bd1-902f-c355ffd4fa70-kube-api-access-6rmfm\") pod \"test-curl-7292-ff3a7e\" (UID: \"8d720a4f-4da7-4bd1-902f-c355ffd4fa70\") " pod="opendatahub/test-curl-7292-ff3a7e" Oct 31 01:20:15.543209 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.543190 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rmfm\" (UniqueName: \"kubernetes.io/projected/8d720a4f-4da7-4bd1-902f-c355ffd4fa70-kube-api-access-6rmfm\") pod \"test-curl-7292-ff3a7e\" (UID: \"8d720a4f-4da7-4bd1-902f-c355ffd4fa70\") " pod="opendatahub/test-curl-7292-ff3a7e" Oct 31 01:20:15.677513 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:15.677459 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-ff3a7e" Oct 31 01:20:16.049184 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:16.049113 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7292-ff3a7e"] Oct 31 01:20:16.056569 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:20:16.056540 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d720a4f_4da7_4bd1_902f_c355ffd4fa70.slice/crio-cd003e004d7783702c791a562caa00ff7e4dda12d19fef217b0df94c841a8b4f WatchSource:0}: Error finding container cd003e004d7783702c791a562caa00ff7e4dda12d19fef217b0df94c841a8b4f: Status 404 returned error can't find the container with id cd003e004d7783702c791a562caa00ff7e4dda12d19fef217b0df94c841a8b4f Oct 31 01:20:16.058732 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:16.058708 5719 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:20:17.068492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:17.068456 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-ff3a7e" event={"ID":"8d720a4f-4da7-4bd1-902f-c355ffd4fa70","Type":"ContainerStarted","Data":"cd003e004d7783702c791a562caa00ff7e4dda12d19fef217b0df94c841a8b4f"} Oct 31 01:20:18.077162 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:18.077123 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7292-ff3a7e" event={"ID":"8d720a4f-4da7-4bd1-902f-c355ffd4fa70","Type":"ContainerStarted","Data":"317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8"} Oct 31 01:20:18.187099 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:18.187055 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7292-ff3a7e" podStartSLOduration=1.708606453 podStartE2EDuration="3.187042053s" totalImagesPullingTime="1.4784356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:18.186648743 +0000 UTC m=+3099.722791052" watchObservedRunningTime="2026-10-31 01:20:18.187042053 +0000 UTC m=+3099.723184357" Oct 31 01:20:19.782650 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:19.782620 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7292-ff3a7e"] Oct 31 01:20:19.787358 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:19.787331 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7292-ff3a7e"] Oct 31 01:20:20.089114 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:20.089078 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7292-ff3a7e" podUID="8d720a4f-4da7-4bd1-902f-c355ffd4fa70" containerName="test-curl-7292-ff3a7e" containerID="cri-o://317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8" gracePeriod=2 Oct 31 01:20:20.138142 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:20:20.138114 5719 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7292-ff3a7e\" is forbidden: User \"system:node:ip-10-0-129-8.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-8.us-west-2.compute.internal' and this object" podUID="8d720a4f-4da7-4bd1-902f-c355ffd4fa70" pod="opendatahub/test-curl-7292-ff3a7e" Oct 31 01:20:22.347520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:22.347457 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-ff3a7e" Oct 31 01:20:22.453475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:22.453445 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d720a4f-4da7-4bd1-902f-c355ffd4fa70-kube-api-access-6rmfm\" (UniqueName: \"kubernetes.io/projected/8d720a4f-4da7-4bd1-902f-c355ffd4fa70-kube-api-access-6rmfm\") pod \"8d720a4f-4da7-4bd1-902f-c355ffd4fa70\" (UID: \"8d720a4f-4da7-4bd1-902f-c355ffd4fa70\") " Oct 31 01:20:22.455664 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:22.455632 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d720a4f-4da7-4bd1-902f-c355ffd4fa70-kube-api-access-6rmfm" pod "8d720a4f-4da7-4bd1-902f-c355ffd4fa70" (UID: "8d720a4f-4da7-4bd1-902f-c355ffd4fa70"). InnerVolumeSpecName "kube-api-access-6rmfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:22.555362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:22.555327 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rmfm\" (UniqueName: \"kubernetes.io/projected/8d720a4f-4da7-4bd1-902f-c355ffd4fa70-kube-api-access-6rmfm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:22.605541 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:22.605469 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d720a4f-4da7-4bd1-902f-c355ffd4fa70" path="/var/lib/kubelet/pods/8d720a4f-4da7-4bd1-902f-c355ffd4fa70/volumes" Oct 31 01:20:23.110390 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:23.110356 5719 generic.go:356] "Generic (PLEG): container finished" podID="8d720a4f-4da7-4bd1-902f-c355ffd4fa70" containerID="317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8" exitCode=137 Oct 31 01:20:23.110565 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:23.110396 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7292-ff3a7e" Oct 31 01:20:23.110565 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:23.110455 5719 scope.go:122] "RemoveContainer" containerID="317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8" Oct 31 01:20:23.119827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:23.119798 5719 scope.go:122] "RemoveContainer" containerID="317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8" Oct 31 01:20:23.120095 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:20:23.120069 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8\": container with ID starting with 317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8 not found: ID does not exist" containerID="317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8" Oct 31 01:20:23.120175 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:23.120104 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8"} err="failed to get container status \"317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8\": rpc error: code = NotFound desc = could not find container \"317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8\": container with ID starting with 317afb3ca0ca753d90eef1419263438c2d9c65cf81b0aff937f753abad904ff8 not found: ID does not exist" Oct 31 01:20:24.003076 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.003035 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" podUID="61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" containerName="main" containerID="cri-o://4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76" gracePeriod=45 Oct 31 01:20:24.285411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.285386 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:20:24.373600 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.373577 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tls-certs\" (UniqueName: \"kubernetes.io/secret/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tls-certs\") pod \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " Oct 31 01:20:24.373769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.373618 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-home\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-home\") pod \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " Oct 31 01:20:24.373769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.373639 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-dshm\") pod \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " Oct 31 01:20:24.373769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.373662 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tmp-dir\") pod \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " Oct 31 01:20:24.373957 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.373774 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-kube-api-access-vc52j\" (UniqueName: \"kubernetes.io/projected/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-kube-api-access-vc52j\") pod \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " Oct 31 01:20:24.373957 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.373844 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-model-cache\") pod \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\" (UID: \"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae\") " Oct 31 01:20:24.373957 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.373903 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-home" pod "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" (UID: "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:24.373957 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.373943 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tmp-dir" pod "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" (UID: "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:24.374164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.374136 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-model-cache" pod "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" (UID: "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:24.374594 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.374571 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:24.374729 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.374600 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:24.374729 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.374638 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:24.375955 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.375931 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tls-certs" pod "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" (UID: "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:24.376041 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.375931 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-dshm" pod "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" (UID: "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:24.376041 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.375984 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-kube-api-access-vc52j" pod "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" (UID: "61a03f47-6bc3-4449-b7b6-fcbebf2b3bae"). InnerVolumeSpecName "kube-api-access-vc52j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:24.475455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.475424 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:24.475611 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.475460 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc52j\" (UniqueName: \"kubernetes.io/projected/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-kube-api-access-vc52j\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:24.475611 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:24.475479 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:25.127348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.127311 5719 generic.go:356] "Generic (PLEG): container finished" podID="61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" containerID="4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76" exitCode=0 Oct 31 01:20:25.127892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.127380 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" Oct 31 01:20:25.127892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.127397 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" event={"ID":"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae","Type":"ContainerDied","Data":"4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76"} Oct 31 01:20:25.127892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.127444 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9" event={"ID":"61a03f47-6bc3-4449-b7b6-fcbebf2b3bae","Type":"ContainerDied","Data":"bd0b6985c040cdb8d67ab5684e1d043bb6a23edbc521ea5ee97c72b6474feae9"} Oct 31 01:20:25.127892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.127465 5719 scope.go:122] "RemoveContainer" containerID="4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76" Oct 31 01:20:25.137176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.137156 5719 scope.go:122] "RemoveContainer" containerID="4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76" Oct 31 01:20:25.137479 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:20:25.137459 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76\": container with ID starting with 4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76 not found: ID does not exist" containerID="4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76" Oct 31 01:20:25.137536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.137487 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76"} err="failed to get container status \"4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76\": rpc error: code = NotFound desc = could not find container \"4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76\": container with ID starting with 4c1f1b4b7c23ad2f6f4d27891b1f7cbd65d979e290d5192952da2d2461d7dc76 not found: ID does not exist" Oct 31 01:20:25.285647 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.285613 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9"] Oct 31 01:20:25.333189 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:25.333158 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-a72754/auth-test-model-a72754-kserve-7f4556749c-k24t9"] Oct 31 01:20:26.602871 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:26.602837 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" path="/var/lib/kubelet/pods/61a03f47-6bc3-4449-b7b6-fcbebf2b3bae/volumes" Oct 31 01:20:33.010956 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:20:33.010925 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:20:45.278611 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:45.278580 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:20:50.170503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.170469 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n"] Oct 31 01:20:50.171012 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.170997 5719 cpu_manager.go:414] "containerMap: removing container" podUID="8d720a4f-4da7-4bd1-902f-c355ffd4fa70" containerName="test-curl-7292-ff3a7e" Oct 31 01:20:50.171057 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.171013 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d720a4f-4da7-4bd1-902f-c355ffd4fa70" containerName="test-curl-7292-ff3a7e" Oct 31 01:20:50.171057 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.171037 5719 cpu_manager.go:414] "containerMap: removing container" podUID="61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" containerName="main" Oct 31 01:20:50.171057 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.171042 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" containerName="main" Oct 31 01:20:50.171156 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.171136 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d720a4f-4da7-4bd1-902f-c355ffd4fa70" containerName="test-curl-7292-ff3a7e" Oct 31 01:20:50.171156 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.171147 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a03f47-6bc3-4449-b7b6-fcbebf2b3bae" containerName="main" Oct 31 01:20:50.175760 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.175744 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.227846 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.227799 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:20:50.275714 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.275663 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:20:50.275714 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.275673 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:50.275912 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.275807 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:20:50.278954 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.278932 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n"] Oct 31 01:20:50.338487 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.338462 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.338643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.338501 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rh6\" (UniqueName: \"kubernetes.io/projected/c2e04609-3833-4ec5-8939-b37930fa384f-kube-api-access-95rh6\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.338643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.338604 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.439800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.439743 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.439800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.439777 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rh6\" (UniqueName: \"kubernetes.io/projected/c2e04609-3833-4ec5-8939-b37930fa384f-kube-api-access-95rh6\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.439946 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.439819 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.440324 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.440297 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.442103 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.442079 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.498385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.498361 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rh6\" (UniqueName: \"kubernetes.io/projected/c2e04609-3833-4ec5-8939-b37930fa384f-kube-api-access-95rh6\") pod \"maas-api-revoke-sub-ed46fa73-46x7n\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:50.789404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:50.789347 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:51.165651 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:51.165620 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n"] Oct 31 01:20:51.171631 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:20:51.171603 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e04609_3833_4ec5_8939_b37930fa384f.slice/crio-d28dcc50d0718c6fcbac12c17dbe00d030b47052b503f9a488eb2026518958a4 WatchSource:0}: Error finding container d28dcc50d0718c6fcbac12c17dbe00d030b47052b503f9a488eb2026518958a4: Status 404 returned error can't find the container with id d28dcc50d0718c6fcbac12c17dbe00d030b47052b503f9a488eb2026518958a4 Oct 31 01:20:51.287379 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:51.287348 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" event={"ID":"c2e04609-3833-4ec5-8939-b37930fa384f","Type":"ContainerStarted","Data":"d28dcc50d0718c6fcbac12c17dbe00d030b47052b503f9a488eb2026518958a4"} Oct 31 01:20:52.295263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:52.295235 5719 generic.go:356] "Generic (PLEG): container finished" podID="c2e04609-3833-4ec5-8939-b37930fa384f" containerID="be2dc374b8718b86bbdf3a41e9b0a7dba4d26678d974899d95b4d49632e0d0fe" exitCode=0 Oct 31 01:20:52.295610 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:52.295327 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" event={"ID":"c2e04609-3833-4ec5-8939-b37930fa384f","Type":"ContainerDied","Data":"be2dc374b8718b86bbdf3a41e9b0a7dba4d26678d974899d95b4d49632e0d0fe"} Oct 31 01:20:53.442893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.442873 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:53.469243 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.469220 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-xks-ca\") pod \"c2e04609-3833-4ec5-8939-b37930fa384f\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " Oct 31 01:20:53.469366 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.469270 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-openshift-ca\") pod \"c2e04609-3833-4ec5-8939-b37930fa384f\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " Oct 31 01:20:53.469366 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.469349 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e04609-3833-4ec5-8939-b37930fa384f-kube-api-access-95rh6\" (UniqueName: \"kubernetes.io/projected/c2e04609-3833-4ec5-8939-b37930fa384f-kube-api-access-95rh6\") pod \"c2e04609-3833-4ec5-8939-b37930fa384f\" (UID: \"c2e04609-3833-4ec5-8939-b37930fa384f\") " Oct 31 01:20:53.469674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.469647 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-openshift-ca" pod "c2e04609-3833-4ec5-8939-b37930fa384f" (UID: "c2e04609-3833-4ec5-8939-b37930fa384f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:53.471267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.471241 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-xks-ca" pod "c2e04609-3833-4ec5-8939-b37930fa384f" (UID: "c2e04609-3833-4ec5-8939-b37930fa384f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:53.471560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.471536 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e04609-3833-4ec5-8939-b37930fa384f-kube-api-access-95rh6" pod "c2e04609-3833-4ec5-8939-b37930fa384f" (UID: "c2e04609-3833-4ec5-8939-b37930fa384f"). InnerVolumeSpecName "kube-api-access-95rh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:53.570421 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.570403 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:53.570536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.570425 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e04609-3833-4ec5-8939-b37930fa384f-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:53.570536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:53.570434 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rh6\" (UniqueName: \"kubernetes.io/projected/c2e04609-3833-4ec5-8939-b37930fa384f-kube-api-access-95rh6\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:54.309630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:54.309601 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" Oct 31 01:20:54.309823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:54.309598 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n" event={"ID":"c2e04609-3833-4ec5-8939-b37930fa384f","Type":"ContainerDied","Data":"d28dcc50d0718c6fcbac12c17dbe00d030b47052b503f9a488eb2026518958a4"} Oct 31 01:20:54.309823 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:20:54.309742 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d28dcc50d0718c6fcbac12c17dbe00d030b47052b503f9a488eb2026518958a4" Oct 31 01:21:00.906198 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.906163 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn"] Oct 31 01:21:00.906712 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.906684 5719 cpu_manager.go:414] "containerMap: removing container" podUID="c2e04609-3833-4ec5-8939-b37930fa384f" containerName="revoke-keys" Oct 31 01:21:00.906765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.906713 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e04609-3833-4ec5-8939-b37930fa384f" containerName="revoke-keys" Oct 31 01:21:00.906842 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.906833 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e04609-3833-4ec5-8939-b37930fa384f" containerName="revoke-keys" Oct 31 01:21:00.910057 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.910041 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:00.939433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.939406 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:00.939554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.939513 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbffw\" (UniqueName: \"kubernetes.io/projected/e8165411-d79e-4415-9f8a-3cfa6ede68d5-kube-api-access-pbffw\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:00.939610 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.939555 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:00.962886 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:00.962861 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:01.010592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.010569 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:21:01.010874 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.010845 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:21:01.011048 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.011030 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:21:01.013362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.013342 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn"] Oct 31 01:21:01.041051 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.041028 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbffw\" (UniqueName: \"kubernetes.io/projected/e8165411-d79e-4415-9f8a-3cfa6ede68d5-kube-api-access-pbffw\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:01.041151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.041067 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:01.041213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.041197 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:01.041863 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.041837 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:01.043705 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.043671 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:01.103405 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.103381 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbffw\" (UniqueName: \"kubernetes.io/projected/e8165411-d79e-4415-9f8a-3cfa6ede68d5-kube-api-access-pbffw\") pod \"maas-api-revoke-sub-089afb46-297sn\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:01.223840 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.223777 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:01.603951 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:01.603913 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn"] Oct 31 01:21:01.610739 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:21:01.610711 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8165411_d79e_4415_9f8a_3cfa6ede68d5.slice/crio-91ecad5174f38ee11d7cc8ac5d31d8fa6084b4e52898840c306f0a2bbc407e6f WatchSource:0}: Error finding container 91ecad5174f38ee11d7cc8ac5d31d8fa6084b4e52898840c306f0a2bbc407e6f: Status 404 returned error can't find the container with id 91ecad5174f38ee11d7cc8ac5d31d8fa6084b4e52898840c306f0a2bbc407e6f Oct 31 01:21:02.357804 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:02.357773 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" event={"ID":"e8165411-d79e-4415-9f8a-3cfa6ede68d5","Type":"ContainerStarted","Data":"713aac2bf73c11e9f2b8ce18ea117d898971fcf845fe452b05018f12f8efea48"} Oct 31 01:21:02.358182 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:02.357821 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" event={"ID":"e8165411-d79e-4415-9f8a-3cfa6ede68d5","Type":"ContainerStarted","Data":"91ecad5174f38ee11d7cc8ac5d31d8fa6084b4e52898840c306f0a2bbc407e6f"} Oct 31 01:21:02.469076 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:02.469021 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" podStartSLOduration=2.46900899 podStartE2EDuration="2.46900899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:02.46884626 +0000 UTC m=+3144.004988564" watchObservedRunningTime="2026-10-31 01:21:02.46900899 +0000 UTC m=+3144.005151294" Oct 31 01:21:03.010432 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:03.010392 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:21:03.364793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:03.364765 5719 generic.go:356] "Generic (PLEG): container finished" podID="e8165411-d79e-4415-9f8a-3cfa6ede68d5" containerID="713aac2bf73c11e9f2b8ce18ea117d898971fcf845fe452b05018f12f8efea48" exitCode=0 Oct 31 01:21:03.365163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:03.364809 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" event={"ID":"e8165411-d79e-4415-9f8a-3cfa6ede68d5","Type":"ContainerDied","Data":"713aac2bf73c11e9f2b8ce18ea117d898971fcf845fe452b05018f12f8efea48"} Oct 31 01:21:04.525624 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.525603 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:04.573269 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.573246 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-openshift-ca\") pod \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " Oct 31 01:21:04.573377 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.573335 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8165411-d79e-4415-9f8a-3cfa6ede68d5-kube-api-access-pbffw\" (UniqueName: \"kubernetes.io/projected/e8165411-d79e-4415-9f8a-3cfa6ede68d5-kube-api-access-pbffw\") pod \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " Oct 31 01:21:04.573416 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.573404 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-xks-ca\") pod \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\" (UID: \"e8165411-d79e-4415-9f8a-3cfa6ede68d5\") " Oct 31 01:21:04.573628 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.573607 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-openshift-ca" pod "e8165411-d79e-4415-9f8a-3cfa6ede68d5" (UID: "e8165411-d79e-4415-9f8a-3cfa6ede68d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:04.573841 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.573822 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:04.575439 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.575418 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8165411-d79e-4415-9f8a-3cfa6ede68d5-kube-api-access-pbffw" pod "e8165411-d79e-4415-9f8a-3cfa6ede68d5" (UID: "e8165411-d79e-4415-9f8a-3cfa6ede68d5"). InnerVolumeSpecName "kube-api-access-pbffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:04.575580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.575559 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-xks-ca" pod "e8165411-d79e-4415-9f8a-3cfa6ede68d5" (UID: "e8165411-d79e-4415-9f8a-3cfa6ede68d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:04.675266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.675190 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbffw\" (UniqueName: \"kubernetes.io/projected/e8165411-d79e-4415-9f8a-3cfa6ede68d5-kube-api-access-pbffw\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:04.675266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:04.675215 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8165411-d79e-4415-9f8a-3cfa6ede68d5-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:05.378360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:05.378329 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" Oct 31 01:21:05.378552 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:05.378363 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn" event={"ID":"e8165411-d79e-4415-9f8a-3cfa6ede68d5","Type":"ContainerDied","Data":"91ecad5174f38ee11d7cc8ac5d31d8fa6084b4e52898840c306f0a2bbc407e6f"} Oct 31 01:21:05.378552 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:05.378445 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91ecad5174f38ee11d7cc8ac5d31d8fa6084b4e52898840c306f0a2bbc407e6f" Oct 31 01:21:17.248569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.248497 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w"] Oct 31 01:21:17.249060 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.249046 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e8165411-d79e-4415-9f8a-3cfa6ede68d5" containerName="revoke-keys" Oct 31 01:21:17.249107 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.249061 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8165411-d79e-4415-9f8a-3cfa6ede68d5" containerName="revoke-keys" Oct 31 01:21:17.249185 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.249178 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8165411-d79e-4415-9f8a-3cfa6ede68d5" containerName="revoke-keys" Oct 31 01:21:17.252635 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.252600 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.308354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.308316 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w"] Oct 31 01:21:17.352807 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.352774 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c32349\"/\"kube-root-ca.crt\"" Oct 31 01:21:17.352807 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.352778 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c32349\"/\"openshift-service-ca.crt\"" Oct 31 01:21:17.353127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.353002 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c32349\"/\"default-dockercfg-l2vdj\"" Oct 31 01:21:17.353203 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.353168 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c32349\"/\"noten-model-c32349-kserve-self-signed-certs\"" Oct 31 01:21:17.398732 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.398669 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-home\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.398896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.398791 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wcth\" (UniqueName: \"kubernetes.io/projected/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-kube-api-access-4wcth\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.398896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.398851 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tls-certs\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.399018 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.398999 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-dshm\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.399159 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.399123 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-model-cache\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.399240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.399170 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tmp-dir\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.500258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.500165 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wcth\" (UniqueName: \"kubernetes.io/projected/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-kube-api-access-4wcth\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.500258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.500245 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tls-certs\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.500457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.500359 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-dshm\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.500457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.500452 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-model-cache\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.500541 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.500486 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tmp-dir\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.500541 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.500521 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-home\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.500948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.500922 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-home\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.501090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.500985 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-model-cache\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.501148 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.501129 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tmp-dir\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.503500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.503476 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-dshm\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.504046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.504025 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tls-certs\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.561433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.561406 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wcth\" (UniqueName: \"kubernetes.io/projected/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-kube-api-access-4wcth\") pod \"noten-model-c32349-kserve-7945c98789-p6x4w\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.565187 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.565159 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:17.968498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:17.968464 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w"] Oct 31 01:21:17.976424 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:21:17.975858 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba25d5ce_e9f5_4bed_ab1f_de193cdc9ddd.slice/crio-b53e0e4db169667955f0617388ab68b21064a60861f7d6518f6d0c04f0b832f7 WatchSource:0}: Error finding container b53e0e4db169667955f0617388ab68b21064a60861f7d6518f6d0c04f0b832f7: Status 404 returned error can't find the container with id b53e0e4db169667955f0617388ab68b21064a60861f7d6518f6d0c04f0b832f7 Oct 31 01:21:18.461167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:18.461134 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" event={"ID":"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd","Type":"ContainerStarted","Data":"574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9"} Oct 31 01:21:18.461574 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:18.461176 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" event={"ID":"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd","Type":"ContainerStarted","Data":"b53e0e4db169667955f0617388ab68b21064a60861f7d6518f6d0c04f0b832f7"} Oct 31 01:21:18.565246 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:18.565217 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:18.575273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:18.575231 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" podStartSLOduration=1.5752191180000001 podStartE2EDuration="1.575219118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:18.573314576 +0000 UTC m=+3160.109456882" watchObservedRunningTime="2026-10-31 01:21:18.575219118 +0000 UTC m=+3160.111361422" Oct 31 01:21:22.492676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:22.492645 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7280-5cdc18"] Oct 31 01:21:22.496584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:22.496558 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:22.547818 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:22.547795 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7280-5cdc18"] Oct 31 01:21:22.592764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:22.592739 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8gxx2\"" Oct 31 01:21:22.653580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:22.653543 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvxp\" (UniqueName: \"kubernetes.io/projected/794f157a-6b49-48c3-94ce-49857e5d7335-kube-api-access-2jvxp\") pod \"test-curl-7280-5cdc18\" (UID: \"794f157a-6b49-48c3-94ce-49857e5d7335\") " pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:22.754862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:22.754797 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvxp\" (UniqueName: \"kubernetes.io/projected/794f157a-6b49-48c3-94ce-49857e5d7335-kube-api-access-2jvxp\") pod \"test-curl-7280-5cdc18\" (UID: \"794f157a-6b49-48c3-94ce-49857e5d7335\") " pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:22.813744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:22.813721 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvxp\" (UniqueName: \"kubernetes.io/projected/794f157a-6b49-48c3-94ce-49857e5d7335-kube-api-access-2jvxp\") pod \"test-curl-7280-5cdc18\" (UID: \"794f157a-6b49-48c3-94ce-49857e5d7335\") " pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:23.109786 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.109755 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:23.484582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.484508 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7280-5cdc18"] Oct 31 01:21:23.491861 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:21:23.491835 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod794f157a_6b49_48c3_94ce_49857e5d7335.slice/crio-d139455c45e47d03f4f34347ba0fe2acf1a60c285e2f682cb5189917870dc03a WatchSource:0}: Error finding container d139455c45e47d03f4f34347ba0fe2acf1a60c285e2f682cb5189917870dc03a: Status 404 returned error can't find the container with id d139455c45e47d03f4f34347ba0fe2acf1a60c285e2f682cb5189917870dc03a Oct 31 01:21:23.552922 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.552896 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24"] Oct 31 01:21:23.557013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.556991 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.611607 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.611585 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24"] Oct 31 01:21:23.653201 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.653176 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:23.653316 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.653258 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:21:23.656258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.656239 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:21:23.656258 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.656238 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:21:23.663768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.663751 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.663896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.663883 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fp8\" (UniqueName: \"kubernetes.io/projected/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-kube-api-access-k9fp8\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.663993 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.663974 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.764599 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.764540 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.764871 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.764838 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fp8\" (UniqueName: \"kubernetes.io/projected/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-kube-api-access-k9fp8\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.764996 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.764924 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.765524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.765499 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.766959 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.766942 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.823748 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.823723 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fp8\" (UniqueName: \"kubernetes.io/projected/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-kube-api-access-k9fp8\") pod \"maas-api-revoke-sub-20ce3765-tqg24\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:23.870095 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:23.870071 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:24.246065 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:24.246035 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24"] Oct 31 01:21:24.252586 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:21:24.252562 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d05cbf7_43e9_4d26_a7b1_fd0b988d8533.slice/crio-0d0f9f785a17dd9134e81cfa07f6c58d6995df44bcd089be2bbec67ce48d9953 WatchSource:0}: Error finding container 0d0f9f785a17dd9134e81cfa07f6c58d6995df44bcd089be2bbec67ce48d9953: Status 404 returned error can't find the container with id 0d0f9f785a17dd9134e81cfa07f6c58d6995df44bcd089be2bbec67ce48d9953 Oct 31 01:21:24.501521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:24.501437 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" event={"ID":"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533","Type":"ContainerStarted","Data":"0d0f9f785a17dd9134e81cfa07f6c58d6995df44bcd089be2bbec67ce48d9953"} Oct 31 01:21:24.503266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:24.503239 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7280-5cdc18" event={"ID":"794f157a-6b49-48c3-94ce-49857e5d7335","Type":"ContainerStarted","Data":"cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed"} Oct 31 01:21:24.503402 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:24.503281 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7280-5cdc18" event={"ID":"794f157a-6b49-48c3-94ce-49857e5d7335","Type":"ContainerStarted","Data":"d139455c45e47d03f4f34347ba0fe2acf1a60c285e2f682cb5189917870dc03a"} Oct 31 01:21:24.613313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:24.613269 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7280-5cdc18" podStartSLOduration=2.613256454 podStartE2EDuration="2.613256454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:24.611554288 +0000 UTC m=+3166.147696594" watchObservedRunningTime="2026-10-31 01:21:24.613256454 +0000 UTC m=+3166.149398758" Oct 31 01:21:25.511140 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:25.511070 5719 generic.go:356] "Generic (PLEG): container finished" podID="1d05cbf7-43e9-4d26-a7b1-fd0b988d8533" containerID="2cedbcb7c1ebaa88b7d196f7608c29c6c85680b548f19910c2691334b1b5ac90" exitCode=0 Oct 31 01:21:25.511296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:25.511159 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" event={"ID":"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533","Type":"ContainerDied","Data":"2cedbcb7c1ebaa88b7d196f7608c29c6c85680b548f19910c2691334b1b5ac90"} Oct 31 01:21:26.089946 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.089912 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7280-5cdc18"] Oct 31 01:21:26.094488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.094458 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7280-5cdc18"] Oct 31 01:21:26.371392 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.371325 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv"] Oct 31 01:21:26.371862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.371847 5719 cpu_manager.go:414] "containerMap: removing container" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" containerName="test-curl-7280-5cdc18" Oct 31 01:21:26.371915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.371862 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" containerName="test-curl-7280-5cdc18" Oct 31 01:21:26.371980 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.371970 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" containerName="test-curl-7280-5cdc18" Oct 31 01:21:26.375294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.375279 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.430247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.430227 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv"] Oct 31 01:21:26.492740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.492681 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zmdb\" (UniqueName: \"kubernetes.io/projected/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-kube-api-access-2zmdb\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.492885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.492845 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.492945 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.492914 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.517514 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.517476 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7280-5cdc18" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" containerName="test-curl-7280-5cdc18" containerID="cri-o://cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed" gracePeriod=2 Oct 31 01:21:26.567013 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:26.566976 5719 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-5cdc18\" is forbidden: User \"system:node:ip-10-0-129-8.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-8.us-west-2.compute.internal' and this object" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:26.593603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.593572 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.593784 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.593677 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zmdb\" (UniqueName: \"kubernetes.io/projected/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-kube-api-access-2zmdb\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.594358 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.593804 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.594652 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.594632 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.607945 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.607150 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.653083 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.653063 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zmdb\" (UniqueName: \"kubernetes.io/projected/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-kube-api-access-2zmdb\") pod \"maas-api-revoke-sub-4287e6ff-9vdkv\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.671386 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.671369 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:26.688340 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.688316 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:26.796403 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.796364 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-xks-ca\") pod \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " Oct 31 01:21:26.796609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.796462 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-openshift-ca\") pod \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " Oct 31 01:21:26.796609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.796536 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-kube-api-access-k9fp8\" (UniqueName: \"kubernetes.io/projected/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-kube-api-access-k9fp8\") pod \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\" (UID: \"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533\") " Oct 31 01:21:26.796902 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.796878 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-openshift-ca" pod "1d05cbf7-43e9-4d26-a7b1-fd0b988d8533" (UID: "1d05cbf7-43e9-4d26-a7b1-fd0b988d8533"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:26.797216 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.797158 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:26.798663 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.798639 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-xks-ca" pod "1d05cbf7-43e9-4d26-a7b1-fd0b988d8533" (UID: "1d05cbf7-43e9-4d26-a7b1-fd0b988d8533"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:26.798771 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.798716 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-kube-api-access-k9fp8" pod "1d05cbf7-43e9-4d26-a7b1-fd0b988d8533" (UID: "1d05cbf7-43e9-4d26-a7b1-fd0b988d8533"). InnerVolumeSpecName "kube-api-access-k9fp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:26.898641 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.898575 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9fp8\" (UniqueName: \"kubernetes.io/projected/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-kube-api-access-k9fp8\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:26.898641 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:26.898601 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:27.082498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.082472 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv"] Oct 31 01:21:27.088649 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:21:27.088613 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84e9dfe_0fc6_4e51_98e2_bf49beeace1a.slice/crio-ec2bf5a732246e22dd1f8fb402f9cf2529723ec7b8dd32bc46f74577d96f3452 WatchSource:0}: Error finding container ec2bf5a732246e22dd1f8fb402f9cf2529723ec7b8dd32bc46f74577d96f3452: Status 404 returned error can't find the container with id ec2bf5a732246e22dd1f8fb402f9cf2529723ec7b8dd32bc46f74577d96f3452 Oct 31 01:21:27.525333 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.525300 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" event={"ID":"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a","Type":"ContainerStarted","Data":"c704ce5bef67a5182892b56901b8ac1eabfb609ed15e144fd88f2ab110b47513"} Oct 31 01:21:27.525716 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.525361 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" event={"ID":"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a","Type":"ContainerStarted","Data":"ec2bf5a732246e22dd1f8fb402f9cf2529723ec7b8dd32bc46f74577d96f3452"} Oct 31 01:21:27.526665 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.526641 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" event={"ID":"1d05cbf7-43e9-4d26-a7b1-fd0b988d8533","Type":"ContainerDied","Data":"0d0f9f785a17dd9134e81cfa07f6c58d6995df44bcd089be2bbec67ce48d9953"} Oct 31 01:21:27.526768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.526705 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d0f9f785a17dd9134e81cfa07f6c58d6995df44bcd089be2bbec67ce48d9953" Oct 31 01:21:27.526768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.526707 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24" Oct 31 01:21:27.566153 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.566128 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:27.578377 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.578325 5719 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:27.685092 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:27.685054 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" podStartSLOduration=1.6850433 podStartE2EDuration="1.6850433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:27.637582567 +0000 UTC m=+3169.173724871" watchObservedRunningTime="2026-10-31 01:21:27.6850433 +0000 UTC m=+3169.221185604" Oct 31 01:21:28.534291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:28.534261 5719 generic.go:356] "Generic (PLEG): container finished" podID="c84e9dfe-0fc6-4e51-98e2-bf49beeace1a" containerID="c704ce5bef67a5182892b56901b8ac1eabfb609ed15e144fd88f2ab110b47513" exitCode=0 Oct 31 01:21:28.534658 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:28.534401 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" event={"ID":"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a","Type":"ContainerDied","Data":"c704ce5bef67a5182892b56901b8ac1eabfb609ed15e144fd88f2ab110b47513"} Oct 31 01:21:28.545889 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:28.545863 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:21:28.768193 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:28.768167 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:28.863711 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:28.863667 5719 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-5cdc18\" is forbidden: User \"system:node:ip-10-0-129-8.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-8.us-west-2.compute.internal' and this object" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:28.921013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:28.920982 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794f157a-6b49-48c3-94ce-49857e5d7335-kube-api-access-2jvxp\" (UniqueName: \"kubernetes.io/projected/794f157a-6b49-48c3-94ce-49857e5d7335-kube-api-access-2jvxp\") pod \"794f157a-6b49-48c3-94ce-49857e5d7335\" (UID: \"794f157a-6b49-48c3-94ce-49857e5d7335\") " Oct 31 01:21:28.923152 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:28.923124 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794f157a-6b49-48c3-94ce-49857e5d7335-kube-api-access-2jvxp" pod "794f157a-6b49-48c3-94ce-49857e5d7335" (UID: "794f157a-6b49-48c3-94ce-49857e5d7335"). InnerVolumeSpecName "kube-api-access-2jvxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:29.021910 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.021885 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvxp\" (UniqueName: \"kubernetes.io/projected/794f157a-6b49-48c3-94ce-49857e5d7335-kube-api-access-2jvxp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:29.542938 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.542905 5719 generic.go:356] "Generic (PLEG): container finished" podID="794f157a-6b49-48c3-94ce-49857e5d7335" containerID="cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed" exitCode=137 Oct 31 01:21:29.543376 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.542971 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:29.543376 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.543003 5719 scope.go:122] "RemoveContainer" containerID="cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed" Oct 31 01:21:29.553914 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.553896 5719 scope.go:122] "RemoveContainer" containerID="cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed" Oct 31 01:21:29.554181 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:29.554161 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed\": container with ID starting with cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed not found: ID does not exist" containerID="cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed" Oct 31 01:21:29.554265 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.554187 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed"} err="failed to get container status \"cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed\": rpc error: code = NotFound desc = could not find container \"cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed\": container with ID starting with cdcfb23000b8f5ea272e434ca304b919ba3d18425f057ac2153ea4e8701837ed not found: ID does not exist" Oct 31 01:21:29.592241 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:29.592194 5719 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-5cdc18\" is forbidden: User \"system:node:ip-10-0-129-8.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-8.us-west-2.compute.internal' and this object" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:29.641269 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:29.641237 5719 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-5cdc18\" is forbidden: User \"system:node:ip-10-0-129-8.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-8.us-west-2.compute.internal' and this object" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:29.694535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.694514 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:29.728560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.728536 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-kube-api-access-2zmdb\" (UniqueName: \"kubernetes.io/projected/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-kube-api-access-2zmdb\") pod \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " Oct 31 01:21:29.728672 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.728576 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-openshift-ca\") pod \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " Oct 31 01:21:29.728741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.728681 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-xks-ca\") pod \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\" (UID: \"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a\") " Oct 31 01:21:29.729000 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.728978 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-openshift-ca" pod "c84e9dfe-0fc6-4e51-98e2-bf49beeace1a" (UID: "c84e9dfe-0fc6-4e51-98e2-bf49beeace1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:29.730811 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.730789 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-kube-api-access-2zmdb" pod "c84e9dfe-0fc6-4e51-98e2-bf49beeace1a" (UID: "c84e9dfe-0fc6-4e51-98e2-bf49beeace1a"). InnerVolumeSpecName "kube-api-access-2zmdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:29.730898 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.730789 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-xks-ca" pod "c84e9dfe-0fc6-4e51-98e2-bf49beeace1a" (UID: "c84e9dfe-0fc6-4e51-98e2-bf49beeace1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:29.829997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.829972 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zmdb\" (UniqueName: \"kubernetes.io/projected/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-kube-api-access-2zmdb\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:29.829997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.829994 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:29.830154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:29.830005 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:29.855092 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:29.855062 5719 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-5cdc18\" is forbidden: User \"system:node:ip-10-0-129-8.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-8.us-west-2.compute.internal' and this object" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:30.551818 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:30.551791 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" Oct 31 01:21:30.552256 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:30.551789 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv" event={"ID":"c84e9dfe-0fc6-4e51-98e2-bf49beeace1a","Type":"ContainerDied","Data":"ec2bf5a732246e22dd1f8fb402f9cf2529723ec7b8dd32bc46f74577d96f3452"} Oct 31 01:21:30.552256 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:30.551973 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec2bf5a732246e22dd1f8fb402f9cf2529723ec7b8dd32bc46f74577d96f3452" Oct 31 01:21:30.603218 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:30.603192 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" path="/var/lib/kubelet/pods/794f157a-6b49-48c3-94ce-49857e5d7335/volumes" Oct 31 01:21:30.658787 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:30.658760 5719 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7280-5cdc18\" is forbidden: User \"system:node:ip-10-0-129-8.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-8.us-west-2.compute.internal' and this object" podUID="794f157a-6b49-48c3-94ce-49857e5d7335" pod="opendatahub/test-curl-7280-5cdc18" Oct 31 01:21:33.014102 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:21:33.014064 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:21:34.295321 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.295290 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk"] Oct 31 01:21:34.295901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.295886 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1d05cbf7-43e9-4d26-a7b1-fd0b988d8533" containerName="revoke-keys" Oct 31 01:21:34.295970 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.295903 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d05cbf7-43e9-4d26-a7b1-fd0b988d8533" containerName="revoke-keys" Oct 31 01:21:34.295970 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.295917 5719 cpu_manager.go:414] "containerMap: removing container" podUID="c84e9dfe-0fc6-4e51-98e2-bf49beeace1a" containerName="revoke-keys" Oct 31 01:21:34.295970 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.295923 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84e9dfe-0fc6-4e51-98e2-bf49beeace1a" containerName="revoke-keys" Oct 31 01:21:34.296075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.296000 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d05cbf7-43e9-4d26-a7b1-fd0b988d8533" containerName="revoke-keys" Oct 31 01:21:34.296075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.296014 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84e9dfe-0fc6-4e51-98e2-bf49beeace1a" containerName="revoke-keys" Oct 31 01:21:34.299607 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.299585 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.352129 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.352096 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:21:34.352248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.352155 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:34.352248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.352240 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:21:34.355498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.355473 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk"] Oct 31 01:21:34.371434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.371413 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.371538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.371461 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqkn\" (UniqueName: \"kubernetes.io/projected/7538c012-87e1-4171-88a8-1e7375d6de48-kube-api-access-hqqkn\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.371588 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.371531 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.399840 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.399822 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:21:34.472301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.472273 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.472431 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.472317 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqkn\" (UniqueName: \"kubernetes.io/projected/7538c012-87e1-4171-88a8-1e7375d6de48-kube-api-access-hqqkn\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.472431 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.472346 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.473125 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.473058 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.474829 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.474808 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.530981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.530963 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqkn\" (UniqueName: \"kubernetes.io/projected/7538c012-87e1-4171-88a8-1e7375d6de48-kube-api-access-hqqkn\") pod \"maas-api-revoke-sub-3a067529-b4gzk\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.613245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.613214 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:34.994301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:34.994226 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk"] Oct 31 01:21:35.000766 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:21:35.000730 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7538c012_87e1_4171_88a8_1e7375d6de48.slice/crio-48adfb0a679601e042ba90264177ef44624f23cf1ffc353dd1e9b513b5a2de92 WatchSource:0}: Error finding container 48adfb0a679601e042ba90264177ef44624f23cf1ffc353dd1e9b513b5a2de92: Status 404 returned error can't find the container with id 48adfb0a679601e042ba90264177ef44624f23cf1ffc353dd1e9b513b5a2de92 Oct 31 01:21:35.585958 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:35.585928 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" event={"ID":"7538c012-87e1-4171-88a8-1e7375d6de48","Type":"ContainerStarted","Data":"c37db7c8b49d6814d9100493b2771e4267dea23133d2895c75a1445eb3afd372"} Oct 31 01:21:35.586439 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:35.585984 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" event={"ID":"7538c012-87e1-4171-88a8-1e7375d6de48","Type":"ContainerStarted","Data":"48adfb0a679601e042ba90264177ef44624f23cf1ffc353dd1e9b513b5a2de92"} Oct 31 01:21:35.697417 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:35.697373 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" podStartSLOduration=1.697360575 podStartE2EDuration="1.697360575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:35.697359976 +0000 UTC m=+3177.233502282" watchObservedRunningTime="2026-10-31 01:21:35.697360575 +0000 UTC m=+3177.233502879" Oct 31 01:21:36.595622 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:36.595584 5719 generic.go:356] "Generic (PLEG): container finished" podID="7538c012-87e1-4171-88a8-1e7375d6de48" containerID="c37db7c8b49d6814d9100493b2771e4267dea23133d2895c75a1445eb3afd372" exitCode=0 Oct 31 01:21:36.602833 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:36.602804 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" event={"ID":"7538c012-87e1-4171-88a8-1e7375d6de48","Type":"ContainerDied","Data":"c37db7c8b49d6814d9100493b2771e4267dea23133d2895c75a1445eb3afd372"} Oct 31 01:21:37.103707 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.103670 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl"] Oct 31 01:21:37.107509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.107490 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.163659 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.163635 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl"] Oct 31 01:21:37.202379 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.202346 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.202490 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.202387 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4lw\" (UniqueName: \"kubernetes.io/projected/a47db979-411d-4ed8-8e33-77fe72d1e63e-kube-api-access-8h4lw\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.202490 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.202418 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.303100 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.303076 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.303220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.303107 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4lw\" (UniqueName: \"kubernetes.io/projected/a47db979-411d-4ed8-8e33-77fe72d1e63e-kube-api-access-8h4lw\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.303220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.303127 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.303811 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.303786 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.305582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.305557 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.328411 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.328388 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:21:37.364489 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.364436 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4lw\" (UniqueName: \"kubernetes.io/projected/a47db979-411d-4ed8-8e33-77fe72d1e63e-kube-api-access-8h4lw\") pod \"maas-api-revoke-sub-7a209c12-9r6gl\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.420883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.420856 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:37.739602 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.739579 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:37.795703 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.795666 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl"] Oct 31 01:21:37.801640 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:21:37.801611 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47db979_411d_4ed8_8e33_77fe72d1e63e.slice/crio-5a76ac58f81e3cca9a1f6bfdb817a9d63128be5d862d5eecf699c320f9959a1b WatchSource:0}: Error finding container 5a76ac58f81e3cca9a1f6bfdb817a9d63128be5d862d5eecf699c320f9959a1b: Status 404 returned error can't find the container with id 5a76ac58f81e3cca9a1f6bfdb817a9d63128be5d862d5eecf699c320f9959a1b Oct 31 01:21:37.809846 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.809824 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-openshift-ca\") pod \"7538c012-87e1-4171-88a8-1e7375d6de48\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " Oct 31 01:21:37.809952 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.809940 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-xks-ca\") pod \"7538c012-87e1-4171-88a8-1e7375d6de48\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " Oct 31 01:21:37.809991 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.809972 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7538c012-87e1-4171-88a8-1e7375d6de48-kube-api-access-hqqkn\" (UniqueName: \"kubernetes.io/projected/7538c012-87e1-4171-88a8-1e7375d6de48-kube-api-access-hqqkn\") pod \"7538c012-87e1-4171-88a8-1e7375d6de48\" (UID: \"7538c012-87e1-4171-88a8-1e7375d6de48\") " Oct 31 01:21:37.810113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.810090 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-openshift-ca" pod "7538c012-87e1-4171-88a8-1e7375d6de48" (UID: "7538c012-87e1-4171-88a8-1e7375d6de48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:37.810253 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.810238 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:37.812050 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.812029 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-xks-ca" pod "7538c012-87e1-4171-88a8-1e7375d6de48" (UID: "7538c012-87e1-4171-88a8-1e7375d6de48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:37.812208 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.812190 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7538c012-87e1-4171-88a8-1e7375d6de48-kube-api-access-hqqkn" pod "7538c012-87e1-4171-88a8-1e7375d6de48" (UID: "7538c012-87e1-4171-88a8-1e7375d6de48"). InnerVolumeSpecName "kube-api-access-hqqkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:37.914245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.914167 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqkn\" (UniqueName: \"kubernetes.io/projected/7538c012-87e1-4171-88a8-1e7375d6de48-kube-api-access-hqqkn\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:37.914245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:37.914202 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7538c012-87e1-4171-88a8-1e7375d6de48-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:38.612398 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:38.612360 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" event={"ID":"a47db979-411d-4ed8-8e33-77fe72d1e63e","Type":"ContainerStarted","Data":"46448b549393cbcb379caae6501131109c1a4273b0948a7a697ad4a4ac9abd9c"} Oct 31 01:21:38.612633 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:38.612420 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" event={"ID":"a47db979-411d-4ed8-8e33-77fe72d1e63e","Type":"ContainerStarted","Data":"5a76ac58f81e3cca9a1f6bfdb817a9d63128be5d862d5eecf699c320f9959a1b"} Oct 31 01:21:38.613831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:38.613810 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" event={"ID":"7538c012-87e1-4171-88a8-1e7375d6de48","Type":"ContainerDied","Data":"48adfb0a679601e042ba90264177ef44624f23cf1ffc353dd1e9b513b5a2de92"} Oct 31 01:21:38.613929 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:38.613844 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48adfb0a679601e042ba90264177ef44624f23cf1ffc353dd1e9b513b5a2de92" Oct 31 01:21:38.613929 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:38.613847 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk" Oct 31 01:21:38.724255 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:38.724216 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" podStartSLOduration=1.724205788 podStartE2EDuration="1.724205788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:38.723041079 +0000 UTC m=+3180.259183363" watchObservedRunningTime="2026-10-31 01:21:38.724205788 +0000 UTC m=+3180.260348093" Oct 31 01:21:39.621354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:39.621307 5719 generic.go:356] "Generic (PLEG): container finished" podID="a47db979-411d-4ed8-8e33-77fe72d1e63e" containerID="46448b549393cbcb379caae6501131109c1a4273b0948a7a697ad4a4ac9abd9c" exitCode=0 Oct 31 01:21:39.621809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:39.621369 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" event={"ID":"a47db979-411d-4ed8-8e33-77fe72d1e63e","Type":"ContainerDied","Data":"46448b549393cbcb379caae6501131109c1a4273b0948a7a697ad4a4ac9abd9c"} Oct 31 01:21:40.782927 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.782904 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:40.842742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.842715 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-xks-ca\") pod \"a47db979-411d-4ed8-8e33-77fe72d1e63e\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " Oct 31 01:21:40.842885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.842754 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47db979-411d-4ed8-8e33-77fe72d1e63e-kube-api-access-8h4lw\" (UniqueName: \"kubernetes.io/projected/a47db979-411d-4ed8-8e33-77fe72d1e63e-kube-api-access-8h4lw\") pod \"a47db979-411d-4ed8-8e33-77fe72d1e63e\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " Oct 31 01:21:40.842885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.842815 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-openshift-ca\") pod \"a47db979-411d-4ed8-8e33-77fe72d1e63e\" (UID: \"a47db979-411d-4ed8-8e33-77fe72d1e63e\") " Oct 31 01:21:40.843187 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.843162 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-openshift-ca" pod "a47db979-411d-4ed8-8e33-77fe72d1e63e" (UID: "a47db979-411d-4ed8-8e33-77fe72d1e63e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:40.844920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.844893 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-xks-ca" pod "a47db979-411d-4ed8-8e33-77fe72d1e63e" (UID: "a47db979-411d-4ed8-8e33-77fe72d1e63e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:40.844998 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.844959 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47db979-411d-4ed8-8e33-77fe72d1e63e-kube-api-access-8h4lw" pod "a47db979-411d-4ed8-8e33-77fe72d1e63e" (UID: "a47db979-411d-4ed8-8e33-77fe72d1e63e"). InnerVolumeSpecName "kube-api-access-8h4lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:40.944458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.944402 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:40.944458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.944425 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h4lw\" (UniqueName: \"kubernetes.io/projected/a47db979-411d-4ed8-8e33-77fe72d1e63e-kube-api-access-8h4lw\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:40.944458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:40.944437 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47db979-411d-4ed8-8e33-77fe72d1e63e-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:41.634854 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:41.634822 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" Oct 31 01:21:41.635016 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:41.634829 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl" event={"ID":"a47db979-411d-4ed8-8e33-77fe72d1e63e","Type":"ContainerDied","Data":"5a76ac58f81e3cca9a1f6bfdb817a9d63128be5d862d5eecf699c320f9959a1b"} Oct 31 01:21:41.635016 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:41.634929 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a76ac58f81e3cca9a1f6bfdb817a9d63128be5d862d5eecf699c320f9959a1b" Oct 31 01:21:45.356888 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:21:45.356856 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w"] Oct 31 01:22:00.377730 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.377669 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" podUID="ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" containerName="main" containerID="cri-o://574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9" gracePeriod=45 Oct 31 01:22:00.535442 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.535400 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" podUID="ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.203:8000/ready\": dial tcp 10.133.0.203:8000: connect: connection refused" Oct 31 01:22:00.673724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.673680 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:22:00.751433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.751399 5719 generic.go:356] "Generic (PLEG): container finished" podID="ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" containerID="574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9" exitCode=0 Oct 31 01:22:00.751586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.751495 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" Oct 31 01:22:00.751586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.751530 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" event={"ID":"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd","Type":"ContainerDied","Data":"574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9"} Oct 31 01:22:00.751586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.751572 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w" event={"ID":"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd","Type":"ContainerDied","Data":"b53e0e4db169667955f0617388ab68b21064a60861f7d6518f6d0c04f0b832f7"} Oct 31 01:22:00.751717 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.751595 5719 scope.go:122] "RemoveContainer" containerID="574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9" Oct 31 01:22:00.761044 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.761032 5719 scope.go:122] "RemoveContainer" containerID="574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9" Oct 31 01:22:00.761354 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:22:00.761333 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9\": container with ID starting with 574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9 not found: ID does not exist" containerID="574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9" Oct 31 01:22:00.761429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.761359 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9"} err="failed to get container status \"574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9\": rpc error: code = NotFound desc = could not find container \"574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9\": container with ID starting with 574f9ba153580a9b24b09f836455ac213b2433cb82974e07004ea78d6443f5e9 not found: ID does not exist" Oct 31 01:22:00.856573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.856543 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-kube-api-access-4wcth\" (UniqueName: \"kubernetes.io/projected/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-kube-api-access-4wcth\") pod \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " Oct 31 01:22:00.856791 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.856604 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tmp-dir\") pod \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " Oct 31 01:22:00.856791 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.856712 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tls-certs\") pod \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " Oct 31 01:22:00.856791 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.856758 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-home\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-home\") pod \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " Oct 31 01:22:00.856791 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.856775 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-dshm\") pod \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " Oct 31 01:22:00.857039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.856845 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-model-cache\") pod \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\" (UID: \"ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd\") " Oct 31 01:22:00.857039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.856932 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tmp-dir" pod "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" (UID: "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:00.857039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.856989 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-home" pod "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" (UID: "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:00.857238 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.857127 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-model-cache" pod "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" (UID: "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:00.857303 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.857291 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:00.857398 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.857305 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:00.857398 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.857314 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:00.859038 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.859013 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-dshm" pod "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" (UID: "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:00.859146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.859067 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-kube-api-access-4wcth" pod "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" (UID: "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd"). InnerVolumeSpecName "kube-api-access-4wcth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:00.859146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.859102 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tls-certs" pod "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" (UID: "ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:00.958232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.958168 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wcth\" (UniqueName: \"kubernetes.io/projected/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-kube-api-access-4wcth\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:00.958232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.958195 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:00.958232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:00.958206 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:01.173726 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:01.173685 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w"] Oct 31 01:22:01.178088 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:01.178063 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c32349/noten-model-c32349-kserve-7945c98789-p6x4w"] Oct 31 01:22:02.613973 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:02.613944 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" path="/var/lib/kubelet/pods/ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd/volumes" Oct 31 01:22:03.009656 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:22:03.009579 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:22:33.011151 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:22:33.011110 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:22:39.538730 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.538677 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j"] Oct 31 01:22:39.539589 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539568 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" containerName="main" Oct 31 01:22:39.539589 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539589 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" containerName="main" Oct 31 01:22:39.539827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539605 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7538c012-87e1-4171-88a8-1e7375d6de48" containerName="revoke-keys" Oct 31 01:22:39.539827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539610 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7538c012-87e1-4171-88a8-1e7375d6de48" containerName="revoke-keys" Oct 31 01:22:39.539827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539651 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a47db979-411d-4ed8-8e33-77fe72d1e63e" containerName="revoke-keys" Oct 31 01:22:39.539827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539657 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47db979-411d-4ed8-8e33-77fe72d1e63e" containerName="revoke-keys" Oct 31 01:22:39.539827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539768 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba25d5ce-e9f5-4bed-ab1f-de193cdc9ddd" containerName="main" Oct 31 01:22:39.539827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539779 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7538c012-87e1-4171-88a8-1e7375d6de48" containerName="revoke-keys" Oct 31 01:22:39.539827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.539790 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47db979-411d-4ed8-8e33-77fe72d1e63e" containerName="revoke-keys" Oct 31 01:22:39.542660 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.542637 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:39.604554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.604528 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j"] Oct 31 01:22:39.645118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.645090 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8gvj\" (UniqueName: \"kubernetes.io/projected/572aeed2-eff3-404e-bc84-77a6652f5926-kube-api-access-t8gvj\") pod \"payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j\" (UID: \"572aeed2-eff3-404e-bc84-77a6652f5926\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:39.645261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.645163 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572aeed2-eff3-404e-bc84-77a6652f5926-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j\" (UID: \"572aeed2-eff3-404e-bc84-77a6652f5926\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:39.648827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.648802 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d067bd\"" Oct 31 01:22:39.696508 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.696485 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d067bd-dockercfg-hhg9k\"" Oct 31 01:22:39.746206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.746182 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8gvj\" (UniqueName: \"kubernetes.io/projected/572aeed2-eff3-404e-bc84-77a6652f5926-kube-api-access-t8gvj\") pod \"payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j\" (UID: \"572aeed2-eff3-404e-bc84-77a6652f5926\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:39.746348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.746228 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572aeed2-eff3-404e-bc84-77a6652f5926-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j\" (UID: \"572aeed2-eff3-404e-bc84-77a6652f5926\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:39.746889 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.746872 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572aeed2-eff3-404e-bc84-77a6652f5926-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j\" (UID: \"572aeed2-eff3-404e-bc84-77a6652f5926\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:39.805490 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.805436 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8gvj\" (UniqueName: \"kubernetes.io/projected/572aeed2-eff3-404e-bc84-77a6652f5926-kube-api-access-t8gvj\") pod \"payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j\" (UID: \"572aeed2-eff3-404e-bc84-77a6652f5926\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:39.855534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:39.855511 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:40.252032 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:40.252002 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j"] Oct 31 01:22:40.258267 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:22:40.258240 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572aeed2_eff3_404e_bc84_77a6652f5926.slice/crio-58fa4ae04f76bd5eba5fa18f6577c5b248cf02baf10c31a353251388a938084c WatchSource:0}: Error finding container 58fa4ae04f76bd5eba5fa18f6577c5b248cf02baf10c31a353251388a938084c: Status 404 returned error can't find the container with id 58fa4ae04f76bd5eba5fa18f6577c5b248cf02baf10c31a353251388a938084c Oct 31 01:22:41.013534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:41.013498 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" event={"ID":"572aeed2-eff3-404e-bc84-77a6652f5926","Type":"ContainerStarted","Data":"eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34"} Oct 31 01:22:41.013969 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:41.013552 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" event={"ID":"572aeed2-eff3-404e-bc84-77a6652f5926","Type":"ContainerStarted","Data":"58fa4ae04f76bd5eba5fa18f6577c5b248cf02baf10c31a353251388a938084c"} Oct 31 01:22:41.013969 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:41.013658 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:41.128761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:41.128682 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" podStartSLOduration=2.128670359 podStartE2EDuration="2.128670359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:41.124922348 +0000 UTC m=+3242.661064654" watchObservedRunningTime="2026-10-31 01:22:41.128670359 +0000 UTC m=+3242.664812700" Oct 31 01:22:41.419043 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:41.419012 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k"] Oct 31 01:22:45.520977 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:45.520900 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:22:45.948435 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:45.948403 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r"] Oct 31 01:22:45.948630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:45.948611 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" containerID="cri-o://90e2199281974a65ea24d36023b8311f74fbaf7122e7735371c5c6ffbfe77e49" gracePeriod=30 Oct 31 01:22:46.263662 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.263584 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq"] Oct 31 01:22:46.268671 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.268651 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:46.318392 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.318360 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea64752-c468-45e0-b0ad-f90ee79e40cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq\" (UID: \"eea64752-c468-45e0-b0ad-f90ee79e40cf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:46.318576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.318425 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2bf\" (UniqueName: \"kubernetes.io/projected/eea64752-c468-45e0-b0ad-f90ee79e40cf-kube-api-access-xb2bf\") pod \"payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq\" (UID: \"eea64752-c468-45e0-b0ad-f90ee79e40cf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:46.322644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.322620 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-6b5091-dockercfg-gg5pn\"" Oct 31 01:22:46.323141 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.323117 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6b5091\"" Oct 31 01:22:46.325035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.325007 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq"] Oct 31 01:22:46.420465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.420431 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea64752-c468-45e0-b0ad-f90ee79e40cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq\" (UID: \"eea64752-c468-45e0-b0ad-f90ee79e40cf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:46.420654 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.420496 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2bf\" (UniqueName: \"kubernetes.io/projected/eea64752-c468-45e0-b0ad-f90ee79e40cf-kube-api-access-xb2bf\") pod \"payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq\" (UID: \"eea64752-c468-45e0-b0ad-f90ee79e40cf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:46.421126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.421109 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea64752-c468-45e0-b0ad-f90ee79e40cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq\" (UID: \"eea64752-c468-45e0-b0ad-f90ee79e40cf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:46.481931 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.481901 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2bf\" (UniqueName: \"kubernetes.io/projected/eea64752-c468-45e0-b0ad-f90ee79e40cf-kube-api-access-xb2bf\") pod \"payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq\" (UID: \"eea64752-c468-45e0-b0ad-f90ee79e40cf\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:46.582376 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.582353 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:46.965991 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:46.965904 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq"] Oct 31 01:22:46.971267 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:22:46.971237 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea64752_c468_45e0_b0ad_f90ee79e40cf.slice/crio-30e176d7c11f632b58dff8c9a1bf14e52c1784c047528321954a201189d0678e WatchSource:0}: Error finding container 30e176d7c11f632b58dff8c9a1bf14e52c1784c047528321954a201189d0678e: Status 404 returned error can't find the container with id 30e176d7c11f632b58dff8c9a1bf14e52c1784c047528321954a201189d0678e Oct 31 01:22:47.058254 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:47.058226 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" event={"ID":"eea64752-c468-45e0-b0ad-f90ee79e40cf","Type":"ContainerStarted","Data":"30e176d7c11f632b58dff8c9a1bf14e52c1784c047528321954a201189d0678e"} Oct 31 01:22:48.066452 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:48.066417 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" event={"ID":"eea64752-c468-45e0-b0ad-f90ee79e40cf","Type":"ContainerStarted","Data":"523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f"} Oct 31 01:22:48.066981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:48.066487 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:22:48.180366 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:48.180312 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" podStartSLOduration=2.180296087 podStartE2EDuration="2.180296087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:48.180211168 +0000 UTC m=+3249.716353473" watchObservedRunningTime="2026-10-31 01:22:48.180296087 +0000 UTC m=+3249.716438393" Oct 31 01:22:51.085636 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.085601 5719 generic.go:356] "Generic (PLEG): container finished" podID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerID="90e2199281974a65ea24d36023b8311f74fbaf7122e7735371c5c6ffbfe77e49" exitCode=0 Oct 31 01:22:51.086126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.085666 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" event={"ID":"9d75beef-0864-46f3-9fab-1cc248c196a1","Type":"ContainerDied","Data":"90e2199281974a65ea24d36023b8311f74fbaf7122e7735371c5c6ffbfe77e49"} Oct 31 01:22:51.204520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.204491 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:22:51.275525 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275456 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-socket\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.275525 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275511 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-credential-socket\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.275764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275540 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d75beef-0864-46f3-9fab-1cc248c196a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d75beef-0864-46f3-9fab-1cc248c196a1-istiod-ca-cert\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.275764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275593 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-kube-api-access-mgwld\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-kube-api-access-mgwld\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.275764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275621 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-podinfo\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.275764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275744 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-certs\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.275764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275754 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-socket" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:51.276145 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275815 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-data\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.276145 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275906 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-token\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-token\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.276145 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.275953 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-envoy\") pod \"9d75beef-0864-46f3-9fab-1cc248c196a1\" (UID: \"9d75beef-0864-46f3-9fab-1cc248c196a1\") " Oct 31 01:22:51.276322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.276199 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-certs" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:51.276322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.276222 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-data" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:51.276322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.276248 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-credential-socket" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:51.276488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.276423 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.276488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.276441 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.276488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.276457 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.276488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.276471 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.276742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.276129 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d75beef-0864-46f3-9fab-1cc248c196a1-istiod-ca-cert" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:51.278554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.278530 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-envoy" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:51.278954 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.278928 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-kube-api-access-mgwld" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "kube-api-access-mgwld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:51.279064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.278972 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-token" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:51.279064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.278982 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-podinfo" pod "9d75beef-0864-46f3-9fab-1cc248c196a1" (UID: "9d75beef-0864-46f3-9fab-1cc248c196a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:22:51.377877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.377842 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.377877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.377877 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.378068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.377890 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d75beef-0864-46f3-9fab-1cc248c196a1-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.378068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.377905 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwld\" (UniqueName: \"kubernetes.io/projected/9d75beef-0864-46f3-9fab-1cc248c196a1-kube-api-access-mgwld\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.378068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:51.377920 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d75beef-0864-46f3-9fab-1cc248c196a1-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:52.023685 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:52.023654 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:22:52.093090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:52.093054 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" Oct 31 01:22:52.093586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:52.093083 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r" event={"ID":"9d75beef-0864-46f3-9fab-1cc248c196a1","Type":"ContainerDied","Data":"3c009059304d026c75f2b3c1bc77e14d05cf4ce0059407dea9bfd899057e6459"} Oct 31 01:22:52.093586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:52.093145 5719 scope.go:122] "RemoveContainer" containerID="90e2199281974a65ea24d36023b8311f74fbaf7122e7735371c5c6ffbfe77e49" Oct 31 01:22:52.245851 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:52.245820 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r"] Oct 31 01:22:52.293967 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:52.293900 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b63c24-openshift-default-d8c64855f-vx98r"] Oct 31 01:22:52.617424 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:52.617389 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" path="/var/lib/kubelet/pods/9d75beef-0864-46f3-9fab-1cc248c196a1/volumes" Oct 31 01:22:56.441462 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.441398 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" podUID="e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" containerName="main" containerID="cri-o://d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23" gracePeriod=45 Oct 31 01:22:56.710131 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.710103 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:22:56.831535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.831504 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-kube-api-access-rbh47\" (UniqueName: \"kubernetes.io/projected/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-kube-api-access-rbh47\") pod \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " Oct 31 01:22:56.831741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.831556 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tls-certs\") pod \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " Oct 31 01:22:56.831741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.831608 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-model-cache\") pod \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " Oct 31 01:22:56.831741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.831652 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tmp-dir\") pod \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " Oct 31 01:22:56.831942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.831788 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-home\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-home\") pod \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " Oct 31 01:22:56.831942 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.831868 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-model-cache" pod "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" (UID: "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:56.832083 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.832023 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-home" pod "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" (UID: "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:56.832207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.832078 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tmp-dir" pod "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" (UID: "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:56.832207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.832138 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-dshm\") pod \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\" (UID: \"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2\") " Oct 31 01:22:56.832598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.832578 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:56.832733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.832603 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:56.832733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.832618 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:56.834052 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.834031 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-kube-api-access-rbh47" pod "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" (UID: "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2"). InnerVolumeSpecName "kube-api-access-rbh47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:56.834251 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.834233 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tls-certs" pod "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" (UID: "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:56.834322 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.834289 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-dshm" pod "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" (UID: "e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:56.933976 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.933957 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:56.934072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.933977 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbh47\" (UniqueName: \"kubernetes.io/projected/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-kube-api-access-rbh47\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:56.934072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:56.933987 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:57.127759 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.127731 5719 generic.go:356] "Generic (PLEG): container finished" podID="e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" containerID="d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23" exitCode=0 Oct 31 01:22:57.129836 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.127778 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" Oct 31 01:22:57.129836 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.127782 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" event={"ID":"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2","Type":"ContainerDied","Data":"d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23"} Oct 31 01:22:57.129836 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.127900 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k" event={"ID":"e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2","Type":"ContainerDied","Data":"1b9b4f20814dd07122719a2f91a7f4cb56a4a36d63129b9f6b2f2bccb9e29d23"} Oct 31 01:22:57.129836 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.127923 5719 scope.go:122] "RemoveContainer" containerID="d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23" Oct 31 01:22:57.138583 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.138561 5719 scope.go:122] "RemoveContainer" containerID="d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23" Oct 31 01:22:57.138907 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:22:57.138878 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23\": container with ID starting with d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23 not found: ID does not exist" containerID="d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23" Oct 31 01:22:57.139015 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.138912 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23"} err="failed to get container status \"d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23\": rpc error: code = NotFound desc = could not find container \"d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23\": container with ID starting with d480660a68e3c72c17fe2399ee3333b6ac000e2c70db1ef9c2dcdd6a68a7dd23 not found: ID does not exist" Oct 31 01:22:57.288706 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.288667 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k"] Oct 31 01:22:57.293507 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:57.293481 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4397fc/auth-test-model-4397fc-kserve-645ffdfc65-r6v7k"] Oct 31 01:22:58.614075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:58.614042 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" path="/var/lib/kubelet/pods/e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2/volumes" Oct 31 01:22:59.075644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:22:59.075613 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:23:01.036774 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.036746 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp"] Oct 31 01:23:01.037308 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.037293 5719 cpu_manager.go:414] "containerMap: removing container" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" Oct 31 01:23:01.037353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.037311 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" Oct 31 01:23:01.037353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.037331 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" containerName="main" Oct 31 01:23:01.037353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.037336 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" containerName="main" Oct 31 01:23:01.037457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.037451 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d75beef-0864-46f3-9fab-1cc248c196a1" containerName="istio-proxy" Oct 31 01:23:01.037492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.037460 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ce34e8-a8d0-4c06-ae51-b523cae9a9f2" containerName="main" Oct 31 01:23:01.042203 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.042177 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:01.141151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.141107 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-08fcc4-dockercfg-2s8bj\"" Oct 31 01:23:01.141151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.141117 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-08fcc4\"" Oct 31 01:23:01.146276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.146247 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp"] Oct 31 01:23:01.178557 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.178512 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x847c\" (UniqueName: \"kubernetes.io/projected/22c20645-c2ae-4e73-852b-50932e28eede-kube-api-access-x847c\") pod \"payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp\" (UID: \"22c20645-c2ae-4e73-852b-50932e28eede\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:01.178784 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.178763 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c20645-c2ae-4e73-852b-50932e28eede-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp\" (UID: \"22c20645-c2ae-4e73-852b-50932e28eede\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:01.191334 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.191311 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l"] Oct 31 01:23:01.195326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.195308 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:01.239742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.239717 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l"] Oct 31 01:23:01.279494 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.279463 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l\" (UID: \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:01.279614 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.279582 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x847c\" (UniqueName: \"kubernetes.io/projected/22c20645-c2ae-4e73-852b-50932e28eede-kube-api-access-x847c\") pod \"payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp\" (UID: \"22c20645-c2ae-4e73-852b-50932e28eede\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:01.279735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.279715 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c20645-c2ae-4e73-852b-50932e28eede-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp\" (UID: \"22c20645-c2ae-4e73-852b-50932e28eede\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:01.279814 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.279752 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqwp\" (UniqueName: \"kubernetes.io/projected/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-kube-api-access-jkqwp\") pod \"payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l\" (UID: \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:01.280386 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.280368 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c20645-c2ae-4e73-852b-50932e28eede-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp\" (UID: \"22c20645-c2ae-4e73-852b-50932e28eede\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:01.340845 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.340816 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x847c\" (UniqueName: \"kubernetes.io/projected/22c20645-c2ae-4e73-852b-50932e28eede-kube-api-access-x847c\") pod \"payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp\" (UID: \"22c20645-c2ae-4e73-852b-50932e28eede\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:01.355519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.355494 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:01.380789 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.380752 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqwp\" (UniqueName: \"kubernetes.io/projected/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-kube-api-access-jkqwp\") pod \"payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l\" (UID: \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:01.381068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.380822 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l\" (UID: \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:01.381504 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.381477 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l\" (UID: \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:01.440123 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.440095 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqwp\" (UniqueName: \"kubernetes.io/projected/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-kube-api-access-jkqwp\") pod \"payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l\" (UID: \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:01.508184 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.508162 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:01.728740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.728669 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp"] Oct 31 01:23:01.734847 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:23:01.734817 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c20645_c2ae_4e73_852b_50932e28eede.slice/crio-036226c19dd209ea3e8208a6fc549c0d74bec9a4e2da30090d9bddf190a6c449 WatchSource:0}: Error finding container 036226c19dd209ea3e8208a6fc549c0d74bec9a4e2da30090d9bddf190a6c449: Status 404 returned error can't find the container with id 036226c19dd209ea3e8208a6fc549c0d74bec9a4e2da30090d9bddf190a6c449 Oct 31 01:23:01.882028 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:01.881999 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l"] Oct 31 01:23:01.887488 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:23:01.887463 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda90758_30d5_41f7_b7bb_3ffa1af55fe3.slice/crio-455e04694c3bfef70a1af212c70b3613244fe6df701911b27ffa2f67f1468c2e WatchSource:0}: Error finding container 455e04694c3bfef70a1af212c70b3613244fe6df701911b27ffa2f67f1468c2e: Status 404 returned error can't find the container with id 455e04694c3bfef70a1af212c70b3613244fe6df701911b27ffa2f67f1468c2e Oct 31 01:23:02.161024 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:02.160992 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" event={"ID":"dda90758-30d5-41f7-b7bb-3ffa1af55fe3","Type":"ContainerStarted","Data":"e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e"} Oct 31 01:23:02.161435 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:02.161047 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" event={"ID":"dda90758-30d5-41f7-b7bb-3ffa1af55fe3","Type":"ContainerStarted","Data":"455e04694c3bfef70a1af212c70b3613244fe6df701911b27ffa2f67f1468c2e"} Oct 31 01:23:02.161435 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:02.161074 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:02.162958 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:02.162934 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" event={"ID":"22c20645-c2ae-4e73-852b-50932e28eede","Type":"ContainerStarted","Data":"8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9"} Oct 31 01:23:02.163061 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:02.162982 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" event={"ID":"22c20645-c2ae-4e73-852b-50932e28eede","Type":"ContainerStarted","Data":"036226c19dd209ea3e8208a6fc549c0d74bec9a4e2da30090d9bddf190a6c449"} Oct 31 01:23:02.163061 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:02.163007 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:02.272414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:02.272368 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" podStartSLOduration=1.272355837 podStartE2EDuration="1.272355837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:02.271653247 +0000 UTC m=+3263.807795564" watchObservedRunningTime="2026-10-31 01:23:02.272355837 +0000 UTC m=+3263.808498143" Oct 31 01:23:02.395072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:02.395018 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" podStartSLOduration=2.395004267 podStartE2EDuration="2.395004267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:02.392351273 +0000 UTC m=+3263.928493579" watchObservedRunningTime="2026-10-31 01:23:02.395004267 +0000 UTC m=+3263.931146571" Oct 31 01:23:03.010343 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:23:03.010296 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:23:06.197674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.197642 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f"] Oct 31 01:23:06.201598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.201581 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:06.301543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.301506 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9332a5\"" Oct 31 01:23:06.301732 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.301551 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9332a5-dockercfg-mrpqr\"" Oct 31 01:23:06.305290 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.305271 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f"] Oct 31 01:23:06.332678 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.332646 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vswbr\" (UniqueName: \"kubernetes.io/projected/e7512be5-ee51-4146-ae6b-615dae55118f-kube-api-access-vswbr\") pod \"payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f\" (UID: \"e7512be5-ee51-4146-ae6b-615dae55118f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:06.332973 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.332936 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7512be5-ee51-4146-ae6b-615dae55118f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f\" (UID: \"e7512be5-ee51-4146-ae6b-615dae55118f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:06.352444 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.352421 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4"] Oct 31 01:23:06.356178 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.356160 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:06.394744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.394715 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4"] Oct 31 01:23:06.434148 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.434115 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vswbr\" (UniqueName: \"kubernetes.io/projected/e7512be5-ee51-4146-ae6b-615dae55118f-kube-api-access-vswbr\") pod \"payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f\" (UID: \"e7512be5-ee51-4146-ae6b-615dae55118f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:06.434300 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.434267 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7512be5-ee51-4146-ae6b-615dae55118f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f\" (UID: \"e7512be5-ee51-4146-ae6b-615dae55118f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:06.435033 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.435007 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7512be5-ee51-4146-ae6b-615dae55118f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f\" (UID: \"e7512be5-ee51-4146-ae6b-615dae55118f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:06.493590 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.493537 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vswbr\" (UniqueName: \"kubernetes.io/projected/e7512be5-ee51-4146-ae6b-615dae55118f-kube-api-access-vswbr\") pod \"payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f\" (UID: \"e7512be5-ee51-4146-ae6b-615dae55118f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:06.516575 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.516549 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:06.535469 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.535449 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6d6\" (UniqueName: \"kubernetes.io/projected/a7eb317e-2ade-4284-a448-c0bc8a06f366-kube-api-access-6d6d6\") pod \"payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4\" (UID: \"a7eb317e-2ade-4284-a448-c0bc8a06f366\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:06.535622 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.535596 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb317e-2ade-4284-a448-c0bc8a06f366-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4\" (UID: \"a7eb317e-2ade-4284-a448-c0bc8a06f366\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:06.637186 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.637150 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6d6\" (UniqueName: \"kubernetes.io/projected/a7eb317e-2ade-4284-a448-c0bc8a06f366-kube-api-access-6d6d6\") pod \"payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4\" (UID: \"a7eb317e-2ade-4284-a448-c0bc8a06f366\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:06.637340 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.637280 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb317e-2ade-4284-a448-c0bc8a06f366-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4\" (UID: \"a7eb317e-2ade-4284-a448-c0bc8a06f366\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:06.637941 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.637922 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb317e-2ade-4284-a448-c0bc8a06f366-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4\" (UID: \"a7eb317e-2ade-4284-a448-c0bc8a06f366\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:06.696153 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.696121 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6d6\" (UniqueName: \"kubernetes.io/projected/a7eb317e-2ade-4284-a448-c0bc8a06f366-kube-api-access-6d6d6\") pod \"payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4\" (UID: \"a7eb317e-2ade-4284-a448-c0bc8a06f366\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:06.903986 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.903952 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f"] Oct 31 01:23:06.910233 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:23:06.910211 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7512be5_ee51_4146_ae6b_615dae55118f.slice/crio-4ca4bf49fc9715c6367e73aae029627d8bd9b23e0ab9d5e3a88813a2e3e95ef8 WatchSource:0}: Error finding container 4ca4bf49fc9715c6367e73aae029627d8bd9b23e0ab9d5e3a88813a2e3e95ef8: Status 404 returned error can't find the container with id 4ca4bf49fc9715c6367e73aae029627d8bd9b23e0ab9d5e3a88813a2e3e95ef8 Oct 31 01:23:06.971604 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:06.971583 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:07.196128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:07.196038 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" event={"ID":"e7512be5-ee51-4146-ae6b-615dae55118f","Type":"ContainerStarted","Data":"ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00"} Oct 31 01:23:07.196128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:07.196094 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" event={"ID":"e7512be5-ee51-4146-ae6b-615dae55118f","Type":"ContainerStarted","Data":"4ca4bf49fc9715c6367e73aae029627d8bd9b23e0ab9d5e3a88813a2e3e95ef8"} Oct 31 01:23:07.196354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:07.196133 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:07.309516 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:07.309468 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" podStartSLOduration=1.30945555 podStartE2EDuration="1.30945555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:07.307368301 +0000 UTC m=+3268.843510606" watchObservedRunningTime="2026-10-31 01:23:07.30945555 +0000 UTC m=+3268.845597854" Oct 31 01:23:07.343399 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:07.343368 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4"] Oct 31 01:23:07.349549 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:23:07.349522 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7eb317e_2ade_4284_a448_c0bc8a06f366.slice/crio-d75e14218857e12de1e40ce53d4e4b9edc87ff46b647cc186d6a7d3a970ddfcc WatchSource:0}: Error finding container d75e14218857e12de1e40ce53d4e4b9edc87ff46b647cc186d6a7d3a970ddfcc: Status 404 returned error can't find the container with id d75e14218857e12de1e40ce53d4e4b9edc87ff46b647cc186d6a7d3a970ddfcc Oct 31 01:23:08.204108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:08.204069 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" event={"ID":"a7eb317e-2ade-4284-a448-c0bc8a06f366","Type":"ContainerStarted","Data":"e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5"} Oct 31 01:23:08.204304 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:08.204124 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" event={"ID":"a7eb317e-2ade-4284-a448-c0bc8a06f366","Type":"ContainerStarted","Data":"d75e14218857e12de1e40ce53d4e4b9edc87ff46b647cc186d6a7d3a970ddfcc"} Oct 31 01:23:08.319380 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:08.319330 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" podStartSLOduration=2.319317243 podStartE2EDuration="2.319317243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:08.315389361 +0000 UTC m=+3269.851531667" watchObservedRunningTime="2026-10-31 01:23:08.319317243 +0000 UTC m=+3269.855459547" Oct 31 01:23:09.211016 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:09.210984 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:13.171639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:13.171607 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:23:13.172083 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:13.171730 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:23:18.207560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:18.207534 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:23:20.220219 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:20.220191 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:23:33.009984 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:23:33.009950 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:23:33.627396 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:33.627364 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:23:56.049279 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:56.049244 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd"] Oct 31 01:23:56.096662 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:56.096637 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45404bb8-gt5hd"] Oct 31 01:23:56.603969 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:23:56.603940 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0a0a3d-95fb-4450-be28-28ed4ee5c589" path="/var/lib/kubelet/pods/8f0a0a3d-95fb-4450-be28-28ed4ee5c589/volumes" Oct 31 01:24:03.009834 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:24:03.009797 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:24:07.049261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:07.049229 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6"] Oct 31 01:24:07.096671 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:07.096642 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-917a3e10-6rwc6"] Oct 31 01:24:08.603310 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:08.603280 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d51b923-7c8d-4da4-93b9-cb8d05c1827c" path="/var/lib/kubelet/pods/2d51b923-7c8d-4da4-93b9-cb8d05c1827c/volumes" Oct 31 01:24:14.975060 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:14.975028 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:24:33.010512 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:24:33.010476 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:24:51.926282 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:51.926246 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:24:56.658569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.658531 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx"] Oct 31 01:24:56.663140 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.663116 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.717719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.717677 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx"] Oct 31 01:24:56.761756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.761729 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:24:56.761883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.761799 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:24:56.761883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.761801 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:24:56.761883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.761850 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:24:56.786169 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.786146 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.786344 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.786326 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.786409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.786361 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqklp\" (UniqueName: \"kubernetes.io/projected/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-kube-api-access-mqklp\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.887889 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.887863 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.888009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.887909 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqklp\" (UniqueName: \"kubernetes.io/projected/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-kube-api-access-mqklp\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.888009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.887953 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.888614 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.888596 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.890260 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.890242 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.947192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.947141 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqklp\" (UniqueName: \"kubernetes.io/projected/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-kube-api-access-mqklp\") pod \"maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:56.976569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:56.976549 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:24:57.372118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:57.372080 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx"] Oct 31 01:24:57.378551 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:24:57.378524 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460fa0d9_d41c_4dcc_a62f_89c1697a28d5.slice/crio-90fc8d9e0e40aba593cc63137715ac42622b2e134dcd209d93a1ca4c5eccbf37 WatchSource:0}: Error finding container 90fc8d9e0e40aba593cc63137715ac42622b2e134dcd209d93a1ca4c5eccbf37: Status 404 returned error can't find the container with id 90fc8d9e0e40aba593cc63137715ac42622b2e134dcd209d93a1ca4c5eccbf37 Oct 31 01:24:57.895735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:57.895676 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" event={"ID":"460fa0d9-d41c-4dcc-a62f-89c1697a28d5","Type":"ContainerStarted","Data":"f29f99bf11858cc3058c1ed39792f9b4507a3b6c86bcdb9000f6ca336842f832"} Oct 31 01:24:57.896167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:57.895761 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" event={"ID":"460fa0d9-d41c-4dcc-a62f-89c1697a28d5","Type":"ContainerStarted","Data":"90fc8d9e0e40aba593cc63137715ac42622b2e134dcd209d93a1ca4c5eccbf37"} Oct 31 01:24:58.009860 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:58.009775 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" podStartSLOduration=2.009761352 podStartE2EDuration="2.009761352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:58.00766596 +0000 UTC m=+3379.543808266" watchObservedRunningTime="2026-10-31 01:24:58.009761352 +0000 UTC m=+3379.545903700" Oct 31 01:24:58.903444 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:58.903407 5719 generic.go:356] "Generic (PLEG): container finished" podID="460fa0d9-d41c-4dcc-a62f-89c1697a28d5" containerID="f29f99bf11858cc3058c1ed39792f9b4507a3b6c86bcdb9000f6ca336842f832" exitCode=0 Oct 31 01:24:58.903954 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:24:58.903491 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" event={"ID":"460fa0d9-d41c-4dcc-a62f-89c1697a28d5","Type":"ContainerDied","Data":"f29f99bf11858cc3058c1ed39792f9b4507a3b6c86bcdb9000f6ca336842f832"} Oct 31 01:25:00.072354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.072330 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:25:00.118994 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.118964 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-xks-ca\") pod \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " Oct 31 01:25:00.119150 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.119083 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-openshift-ca\") pod \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " Oct 31 01:25:00.119196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.119150 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-kube-api-access-mqklp\" (UniqueName: \"kubernetes.io/projected/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-kube-api-access-mqklp\") pod \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\" (UID: \"460fa0d9-d41c-4dcc-a62f-89c1697a28d5\") " Oct 31 01:25:00.119543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.119513 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-openshift-ca" pod "460fa0d9-d41c-4dcc-a62f-89c1697a28d5" (UID: "460fa0d9-d41c-4dcc-a62f-89c1697a28d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:00.121054 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.121034 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-xks-ca" pod "460fa0d9-d41c-4dcc-a62f-89c1697a28d5" (UID: "460fa0d9-d41c-4dcc-a62f-89c1697a28d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:00.121433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.121408 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-kube-api-access-mqklp" pod "460fa0d9-d41c-4dcc-a62f-89c1697a28d5" (UID: "460fa0d9-d41c-4dcc-a62f-89c1697a28d5"). InnerVolumeSpecName "kube-api-access-mqklp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:00.221026 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.220957 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:00.221026 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.220985 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:00.221026 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.220995 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqklp\" (UniqueName: \"kubernetes.io/projected/460fa0d9-d41c-4dcc-a62f-89c1697a28d5-kube-api-access-mqklp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:00.918853 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.918783 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" Oct 31 01:25:00.918997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.918778 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx" event={"ID":"460fa0d9-d41c-4dcc-a62f-89c1697a28d5","Type":"ContainerDied","Data":"90fc8d9e0e40aba593cc63137715ac42622b2e134dcd209d93a1ca4c5eccbf37"} Oct 31 01:25:00.918997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:00.918908 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90fc8d9e0e40aba593cc63137715ac42622b2e134dcd209d93a1ca4c5eccbf37" Oct 31 01:25:03.008934 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:25:03.008899 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:25:33.010302 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:25:33.010262 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:25:35.242640 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.242606 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v"] Oct 31 01:25:35.243172 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.243157 5719 cpu_manager.go:414] "containerMap: removing container" podUID="460fa0d9-d41c-4dcc-a62f-89c1697a28d5" containerName="revoke-keys" Oct 31 01:25:35.243219 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.243173 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460fa0d9-d41c-4dcc-a62f-89c1697a28d5" containerName="revoke-keys" Oct 31 01:25:35.243296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.243286 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="460fa0d9-d41c-4dcc-a62f-89c1697a28d5" containerName="revoke-keys" Oct 31 01:25:35.246808 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.246787 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.298555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.298531 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:25:35.298744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.298673 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:25:35.346211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.346183 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:25:35.346211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.346183 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:25:35.349047 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.349028 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v"] Oct 31 01:25:35.361742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.361718 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0001331a-9326-4238-8893-9548aa95f208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.361847 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.361827 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjs5s\" (UniqueName: \"kubernetes.io/projected/0001331a-9326-4238-8893-9548aa95f208-kube-api-access-jjs5s\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.361919 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.361905 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0001331a-9326-4238-8893-9548aa95f208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.463352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.463327 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0001331a-9326-4238-8893-9548aa95f208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.463491 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.463379 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0001331a-9326-4238-8893-9548aa95f208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.463491 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.463430 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjs5s\" (UniqueName: \"kubernetes.io/projected/0001331a-9326-4238-8893-9548aa95f208-kube-api-access-jjs5s\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.464115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.464093 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0001331a-9326-4238-8893-9548aa95f208-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.465707 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.465670 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0001331a-9326-4238-8893-9548aa95f208-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.522421 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.522365 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjs5s\" (UniqueName: \"kubernetes.io/projected/0001331a-9326-4238-8893-9548aa95f208-kube-api-access-jjs5s\") pod \"maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.559904 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.559883 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:35.936093 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:35.936051 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v"] Oct 31 01:25:35.942419 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:25:35.942388 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0001331a_9326_4238_8893_9548aa95f208.slice/crio-89997328b24f35fa7c8be259a9ce45e74eb777ba414dd5effe896c12c3a3316f WatchSource:0}: Error finding container 89997328b24f35fa7c8be259a9ce45e74eb777ba414dd5effe896c12c3a3316f: Status 404 returned error can't find the container with id 89997328b24f35fa7c8be259a9ce45e74eb777ba414dd5effe896c12c3a3316f Oct 31 01:25:36.142940 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:36.142909 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" event={"ID":"0001331a-9326-4238-8893-9548aa95f208","Type":"ContainerStarted","Data":"89997328b24f35fa7c8be259a9ce45e74eb777ba414dd5effe896c12c3a3316f"} Oct 31 01:25:37.089783 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.089756 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls"] Oct 31 01:25:37.099849 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.099827 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.149463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.149441 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls"] Oct 31 01:25:37.153372 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.153349 5719 generic.go:356] "Generic (PLEG): container finished" podID="0001331a-9326-4238-8893-9548aa95f208" containerID="665e7dbe24b8c1df96fafcad33d20d2fbafdf5ffcab5d29ba632e0f1e7a7458d" exitCode=0 Oct 31 01:25:37.153524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.153500 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" event={"ID":"0001331a-9326-4238-8893-9548aa95f208","Type":"ContainerDied","Data":"665e7dbe24b8c1df96fafcad33d20d2fbafdf5ffcab5d29ba632e0f1e7a7458d"} Oct 31 01:25:37.180892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.180870 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.181008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.180919 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.181008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.180952 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-272ll\" (UniqueName: \"kubernetes.io/projected/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-kube-api-access-272ll\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.282495 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.282472 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-272ll\" (UniqueName: \"kubernetes.io/projected/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-kube-api-access-272ll\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.282612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.282569 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.282612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.282602 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.283240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.283209 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.285158 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.285137 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.341351 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.341300 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-272ll\" (UniqueName: \"kubernetes.io/projected/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-kube-api-access-272ll\") pod \"maas-api-revoke-keys-e2e-autores-6b5091-f99ls\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.413674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.413650 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:37.802885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:37.802811 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls"] Oct 31 01:25:37.808518 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:25:37.808493 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-389916ccbc165cd292085eec34c7c043b505f2b293d2bdd98b1f28560c5bef71 WatchSource:0}: Error finding container 389916ccbc165cd292085eec34c7c043b505f2b293d2bdd98b1f28560c5bef71: Status 404 returned error can't find the container with id 389916ccbc165cd292085eec34c7c043b505f2b293d2bdd98b1f28560c5bef71 Oct 31 01:25:38.161880 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.161846 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" event={"ID":"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a","Type":"ContainerStarted","Data":"5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3"} Oct 31 01:25:38.162278 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.161899 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" event={"ID":"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a","Type":"ContainerStarted","Data":"389916ccbc165cd292085eec34c7c043b505f2b293d2bdd98b1f28560c5bef71"} Oct 31 01:25:38.317955 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.317929 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:38.396483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.396420 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0001331a-9326-4238-8893-9548aa95f208-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0001331a-9326-4238-8893-9548aa95f208-maas-api-openshift-ca\") pod \"0001331a-9326-4238-8893-9548aa95f208\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " Oct 31 01:25:38.396724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.396483 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0001331a-9326-4238-8893-9548aa95f208-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0001331a-9326-4238-8893-9548aa95f208-maas-api-xks-ca\") pod \"0001331a-9326-4238-8893-9548aa95f208\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " Oct 31 01:25:38.396724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.396513 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0001331a-9326-4238-8893-9548aa95f208-kube-api-access-jjs5s\" (UniqueName: \"kubernetes.io/projected/0001331a-9326-4238-8893-9548aa95f208-kube-api-access-jjs5s\") pod \"0001331a-9326-4238-8893-9548aa95f208\" (UID: \"0001331a-9326-4238-8893-9548aa95f208\") " Oct 31 01:25:38.396872 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.396818 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0001331a-9326-4238-8893-9548aa95f208-maas-api-openshift-ca" pod "0001331a-9326-4238-8893-9548aa95f208" (UID: "0001331a-9326-4238-8893-9548aa95f208"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:38.398770 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.398752 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0001331a-9326-4238-8893-9548aa95f208-maas-api-xks-ca" pod "0001331a-9326-4238-8893-9548aa95f208" (UID: "0001331a-9326-4238-8893-9548aa95f208"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:38.398839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.398822 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0001331a-9326-4238-8893-9548aa95f208-kube-api-access-jjs5s" pod "0001331a-9326-4238-8893-9548aa95f208" (UID: "0001331a-9326-4238-8893-9548aa95f208"). InnerVolumeSpecName "kube-api-access-jjs5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:38.431746 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.431684 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" podStartSLOduration=1.431673367 podStartE2EDuration="1.431673367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:38.273867727 +0000 UTC m=+3419.810010033" watchObservedRunningTime="2026-10-31 01:25:38.431673367 +0000 UTC m=+3419.967815671" Oct 31 01:25:38.497750 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.497726 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0001331a-9326-4238-8893-9548aa95f208-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:38.497750 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.497749 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0001331a-9326-4238-8893-9548aa95f208-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:38.497899 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:38.497759 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjs5s\" (UniqueName: \"kubernetes.io/projected/0001331a-9326-4238-8893-9548aa95f208-kube-api-access-jjs5s\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:39.170860 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:39.170828 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" Oct 31 01:25:39.171268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:39.170860 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v" event={"ID":"0001331a-9326-4238-8893-9548aa95f208","Type":"ContainerDied","Data":"89997328b24f35fa7c8be259a9ce45e74eb777ba414dd5effe896c12c3a3316f"} Oct 31 01:25:39.171268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:39.170913 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89997328b24f35fa7c8be259a9ce45e74eb777ba414dd5effe896c12c3a3316f" Oct 31 01:25:39.172660 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:39.172632 5719 generic.go:356] "Generic (PLEG): container finished" podID="257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a" containerID="5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3" exitCode=0 Oct 31 01:25:39.172899 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:39.172766 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" event={"ID":"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a","Type":"ContainerDied","Data":"5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3"} Oct 31 01:25:40.317128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.317100 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:40.415657 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.415632 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-kube-api-access-272ll\" (UniqueName: \"kubernetes.io/projected/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-kube-api-access-272ll\") pod \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " Oct 31 01:25:40.415828 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.415669 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-xks-ca\") pod \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " Oct 31 01:25:40.415828 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.415736 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-openshift-ca\") pod \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\" (UID: \"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a\") " Oct 31 01:25:40.416118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.416093 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-openshift-ca" pod "257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a" (UID: "257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:40.416220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.416205 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:40.417948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.417916 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-xks-ca" pod "257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a" (UID: "257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:40.418070 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.418001 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-kube-api-access-272ll" pod "257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a" (UID: "257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a"). InnerVolumeSpecName "kube-api-access-272ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:40.517483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.517418 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-272ll\" (UniqueName: \"kubernetes.io/projected/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-kube-api-access-272ll\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:40.517483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:40.517445 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:41.186826 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:41.186789 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" event={"ID":"257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a","Type":"ContainerDied","Data":"389916ccbc165cd292085eec34c7c043b505f2b293d2bdd98b1f28560c5bef71"} Oct 31 01:25:41.186826 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:41.186820 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls" Oct 31 01:25:41.187029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:41.186839 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="389916ccbc165cd292085eec34c7c043b505f2b293d2bdd98b1f28560c5bef71" Oct 31 01:25:41.981030 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:41.981001 5719 scope.go:122] "RemoveContainer" containerID="ff5ad7c75ceb3a3fef88fbfa94c2d6023fd40706d786ad656b09473f053a0024" Oct 31 01:25:41.991001 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:41.990974 5719 scope.go:122] "RemoveContainer" containerID="a60e4b5eeadca7a22553a722fd5511bc4e4cabec4c4bf04f3aaa77788ed0d971" Oct 31 01:25:41.999934 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:41.999915 5719 scope.go:122] "RemoveContainer" containerID="12b721dd56f4a3c47ff225bbe13e5737711f5b0e0d02de0ce48c30c96e4d6a2c" Oct 31 01:25:44.381724 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:25:44.381675 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:25:44.566023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:44.565992 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq"] Oct 31 01:25:49.567143 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:49.567064 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" podUID="eea64752-c468-45e0-b0ad-f90ee79e40cf" containerName="payload-pre-processing" containerID="cri-o://523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f" gracePeriod=25 Oct 31 01:25:49.694388 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:49.694361 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j"] Oct 31 01:25:49.823537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:49.823474 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:25:49.904936 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:49.904905 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea64752-c468-45e0-b0ad-f90ee79e40cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea64752-c468-45e0-b0ad-f90ee79e40cf-plugins-config-volume\") pod \"eea64752-c468-45e0-b0ad-f90ee79e40cf\" (UID: \"eea64752-c468-45e0-b0ad-f90ee79e40cf\") " Oct 31 01:25:49.905086 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:49.905018 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea64752-c468-45e0-b0ad-f90ee79e40cf-kube-api-access-xb2bf\" (UniqueName: \"kubernetes.io/projected/eea64752-c468-45e0-b0ad-f90ee79e40cf-kube-api-access-xb2bf\") pod \"eea64752-c468-45e0-b0ad-f90ee79e40cf\" (UID: \"eea64752-c468-45e0-b0ad-f90ee79e40cf\") " Oct 31 01:25:49.905255 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:49.905231 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea64752-c468-45e0-b0ad-f90ee79e40cf-plugins-config-volume" pod "eea64752-c468-45e0-b0ad-f90ee79e40cf" (UID: "eea64752-c468-45e0-b0ad-f90ee79e40cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:49.905531 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:49.905509 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea64752-c468-45e0-b0ad-f90ee79e40cf-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:49.907342 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:49.907319 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea64752-c468-45e0-b0ad-f90ee79e40cf-kube-api-access-xb2bf" pod "eea64752-c468-45e0-b0ad-f90ee79e40cf" (UID: "eea64752-c468-45e0-b0ad-f90ee79e40cf"). InnerVolumeSpecName "kube-api-access-xb2bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:50.006364 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.006337 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb2bf\" (UniqueName: \"kubernetes.io/projected/eea64752-c468-45e0-b0ad-f90ee79e40cf-kube-api-access-xb2bf\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:50.243594 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.243514 5719 generic.go:356] "Generic (PLEG): container finished" podID="eea64752-c468-45e0-b0ad-f90ee79e40cf" containerID="523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f" exitCode=0 Oct 31 01:25:50.243594 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.243581 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" Oct 31 01:25:50.243809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.243602 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" event={"ID":"eea64752-c468-45e0-b0ad-f90ee79e40cf","Type":"ContainerDied","Data":"523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f"} Oct 31 01:25:50.243809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.243708 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq" event={"ID":"eea64752-c468-45e0-b0ad-f90ee79e40cf","Type":"ContainerDied","Data":"30e176d7c11f632b58dff8c9a1bf14e52c1784c047528321954a201189d0678e"} Oct 31 01:25:50.243809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.243727 5719 scope.go:122] "RemoveContainer" containerID="523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f" Oct 31 01:25:50.254274 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.254256 5719 scope.go:122] "RemoveContainer" containerID="523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f" Oct 31 01:25:50.254542 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:25:50.254521 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f\": container with ID starting with 523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f not found: ID does not exist" containerID="523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f" Oct 31 01:25:50.254624 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.254548 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f"} err="failed to get container status \"523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f\": rpc error: code = NotFound desc = could not find container \"523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f\": container with ID starting with 523a6c5d877169f533a71ef7b8489b731c32024d784a045fec93d72ba3037b3f not found: ID does not exist" Oct 31 01:25:50.403129 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.403103 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq"] Oct 31 01:25:50.407141 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.407115 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6b5091-755985cf47-tf5vq"] Oct 31 01:25:50.602194 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:50.602172 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea64752-c468-45e0-b0ad-f90ee79e40cf" path="/var/lib/kubelet/pods/eea64752-c468-45e0-b0ad-f90ee79e40cf/volumes" Oct 31 01:25:51.339085 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:25:51.339050 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:25:53.304314 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.304287 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb"] Oct 31 01:25:53.304850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.304835 5719 cpu_manager.go:414] "containerMap: removing container" podUID="257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a" containerName="revoke-keys" Oct 31 01:25:53.304906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.304850 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a" containerName="revoke-keys" Oct 31 01:25:53.304906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.304868 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0001331a-9326-4238-8893-9548aa95f208" containerName="revoke-keys" Oct 31 01:25:53.304906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.304873 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0001331a-9326-4238-8893-9548aa95f208" containerName="revoke-keys" Oct 31 01:25:53.304906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.304895 5719 cpu_manager.go:414] "containerMap: removing container" podUID="eea64752-c468-45e0-b0ad-f90ee79e40cf" containerName="payload-pre-processing" Oct 31 01:25:53.304906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.304900 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea64752-c468-45e0-b0ad-f90ee79e40cf" containerName="payload-pre-processing" Oct 31 01:25:53.305066 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.304996 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a" containerName="revoke-keys" Oct 31 01:25:53.305066 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.305007 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0001331a-9326-4238-8893-9548aa95f208" containerName="revoke-keys" Oct 31 01:25:53.305066 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.305017 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea64752-c468-45e0-b0ad-f90ee79e40cf" containerName="payload-pre-processing" Oct 31 01:25:53.309713 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.309673 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.360114 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.360090 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:25:53.360264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.360116 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:25:53.363986 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.363964 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb"] Oct 31 01:25:53.407650 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.407627 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:25:53.407758 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.407665 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:25:53.438472 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.438450 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.438614 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.438596 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzxc\" (UniqueName: \"kubernetes.io/projected/a86b94ab-470a-4de5-bff6-436f45bf6d4c-kube-api-access-pdzxc\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.438753 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.438732 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.539829 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.539808 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.539941 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.539918 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzxc\" (UniqueName: \"kubernetes.io/projected/a86b94ab-470a-4de5-bff6-436f45bf6d4c-kube-api-access-pdzxc\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.540005 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.539956 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.540610 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.540592 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.542292 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.542270 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.603681 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.603652 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzxc\" (UniqueName: \"kubernetes.io/projected/a86b94ab-470a-4de5-bff6-436f45bf6d4c-kube-api-access-pdzxc\") pod \"maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.623583 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.623560 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:53.999732 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:53.999631 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb"] Oct 31 01:25:54.013029 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:25:54.013003 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86b94ab_470a_4de5_bff6_436f45bf6d4c.slice/crio-c86266cd82338697891519e0aaf818df8c0ab20b40e1e485250e959e5354899e WatchSource:0}: Error finding container c86266cd82338697891519e0aaf818df8c0ab20b40e1e485250e959e5354899e: Status 404 returned error can't find the container with id c86266cd82338697891519e0aaf818df8c0ab20b40e1e485250e959e5354899e Oct 31 01:25:54.049584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:54.049555 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n"] Oct 31 01:25:54.054202 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:54.054173 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed46fa73-46x7n"] Oct 31 01:25:54.270268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:54.270210 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" event={"ID":"a86b94ab-470a-4de5-bff6-436f45bf6d4c","Type":"ContainerStarted","Data":"c86266cd82338697891519e0aaf818df8c0ab20b40e1e485250e959e5354899e"} Oct 31 01:25:54.396279 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:25:54.396249 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:25:54.603282 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:54.603255 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e04609-3833-4ec5-8939-b37930fa384f" path="/var/lib/kubelet/pods/c2e04609-3833-4ec5-8939-b37930fa384f/volumes" Oct 31 01:25:54.695076 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:54.695045 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" podUID="572aeed2-eff3-404e-bc84-77a6652f5926" containerName="payload-processing" containerID="cri-o://eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34" gracePeriod=25 Oct 31 01:25:54.955457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:54.955432 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:25:55.056039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.055999 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572aeed2-eff3-404e-bc84-77a6652f5926-kube-api-access-t8gvj\" (UniqueName: \"kubernetes.io/projected/572aeed2-eff3-404e-bc84-77a6652f5926-kube-api-access-t8gvj\") pod \"572aeed2-eff3-404e-bc84-77a6652f5926\" (UID: \"572aeed2-eff3-404e-bc84-77a6652f5926\") " Oct 31 01:25:55.056173 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.056126 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572aeed2-eff3-404e-bc84-77a6652f5926-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572aeed2-eff3-404e-bc84-77a6652f5926-plugins-config-volume\") pod \"572aeed2-eff3-404e-bc84-77a6652f5926\" (UID: \"572aeed2-eff3-404e-bc84-77a6652f5926\") " Oct 31 01:25:55.056527 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.056502 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572aeed2-eff3-404e-bc84-77a6652f5926-plugins-config-volume" pod "572aeed2-eff3-404e-bc84-77a6652f5926" (UID: "572aeed2-eff3-404e-bc84-77a6652f5926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:55.058252 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.058230 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572aeed2-eff3-404e-bc84-77a6652f5926-kube-api-access-t8gvj" pod "572aeed2-eff3-404e-bc84-77a6652f5926" (UID: "572aeed2-eff3-404e-bc84-77a6652f5926"). InnerVolumeSpecName "kube-api-access-t8gvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:55.157585 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.157513 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8gvj\" (UniqueName: \"kubernetes.io/projected/572aeed2-eff3-404e-bc84-77a6652f5926-kube-api-access-t8gvj\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:55.157585 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.157543 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/572aeed2-eff3-404e-bc84-77a6652f5926-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:55.277524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.277491 5719 generic.go:356] "Generic (PLEG): container finished" podID="a86b94ab-470a-4de5-bff6-436f45bf6d4c" containerID="c62de5cf8ba7958be9506c4a35289e40a14f724d38342193c2c91fd556a2891b" exitCode=0 Oct 31 01:25:55.277672 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.277578 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" event={"ID":"a86b94ab-470a-4de5-bff6-436f45bf6d4c","Type":"ContainerDied","Data":"c62de5cf8ba7958be9506c4a35289e40a14f724d38342193c2c91fd556a2891b"} Oct 31 01:25:55.279110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.279085 5719 generic.go:356] "Generic (PLEG): container finished" podID="572aeed2-eff3-404e-bc84-77a6652f5926" containerID="eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34" exitCode=0 Oct 31 01:25:55.279230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.279128 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" Oct 31 01:25:55.279230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.279160 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" event={"ID":"572aeed2-eff3-404e-bc84-77a6652f5926","Type":"ContainerDied","Data":"eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34"} Oct 31 01:25:55.279230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.279201 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j" event={"ID":"572aeed2-eff3-404e-bc84-77a6652f5926","Type":"ContainerDied","Data":"58fa4ae04f76bd5eba5fa18f6577c5b248cf02baf10c31a353251388a938084c"} Oct 31 01:25:55.279230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.279222 5719 scope.go:122] "RemoveContainer" containerID="eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34" Oct 31 01:25:55.289630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.289615 5719 scope.go:122] "RemoveContainer" containerID="eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34" Oct 31 01:25:55.289899 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:25:55.289878 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34\": container with ID starting with eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34 not found: ID does not exist" containerID="eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34" Oct 31 01:25:55.289958 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.289906 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34"} err="failed to get container status \"eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34\": rpc error: code = NotFound desc = could not find container \"eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34\": container with ID starting with eaa03bad0ff34454d78da55709506254efa05680ca279ad4cdc9f95c8bccaa34 not found: ID does not exist" Oct 31 01:25:55.500720 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.500641 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j"] Oct 31 01:25:55.504476 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:55.504453 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d067bd-5f586dc769-w9w7j"] Oct 31 01:25:56.431265 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.431235 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:56.569195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.569165 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a86b94ab-470a-4de5-bff6-436f45bf6d4c-kube-api-access-pdzxc\" (UniqueName: \"kubernetes.io/projected/a86b94ab-470a-4de5-bff6-436f45bf6d4c-kube-api-access-pdzxc\") pod \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " Oct 31 01:25:56.569582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.569279 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-openshift-ca\") pod \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " Oct 31 01:25:56.569582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.569365 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-xks-ca\") pod \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\" (UID: \"a86b94ab-470a-4de5-bff6-436f45bf6d4c\") " Oct 31 01:25:56.569718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.569670 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-openshift-ca" pod "a86b94ab-470a-4de5-bff6-436f45bf6d4c" (UID: "a86b94ab-470a-4de5-bff6-436f45bf6d4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:56.569866 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.569810 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:56.571414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.571388 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86b94ab-470a-4de5-bff6-436f45bf6d4c-kube-api-access-pdzxc" pod "a86b94ab-470a-4de5-bff6-436f45bf6d4c" (UID: "a86b94ab-470a-4de5-bff6-436f45bf6d4c"). InnerVolumeSpecName "kube-api-access-pdzxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:56.571528 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.571495 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-xks-ca" pod "a86b94ab-470a-4de5-bff6-436f45bf6d4c" (UID: "a86b94ab-470a-4de5-bff6-436f45bf6d4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:56.603906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.603880 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572aeed2-eff3-404e-bc84-77a6652f5926" path="/var/lib/kubelet/pods/572aeed2-eff3-404e-bc84-77a6652f5926/volumes" Oct 31 01:25:56.671037 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.671019 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdzxc\" (UniqueName: \"kubernetes.io/projected/a86b94ab-470a-4de5-bff6-436f45bf6d4c-kube-api-access-pdzxc\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:56.671037 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:56.671037 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a86b94ab-470a-4de5-bff6-436f45bf6d4c-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:25:57.294163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:57.294133 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" Oct 31 01:25:57.294382 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:57.294131 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb" event={"ID":"a86b94ab-470a-4de5-bff6-436f45bf6d4c","Type":"ContainerDied","Data":"c86266cd82338697891519e0aaf818df8c0ab20b40e1e485250e959e5354899e"} Oct 31 01:25:57.294382 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:25:57.294270 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86266cd82338697891519e0aaf818df8c0ab20b40e1e485250e959e5354899e" Oct 31 01:26:01.035723 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.035672 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv"] Oct 31 01:26:01.036259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.036244 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a86b94ab-470a-4de5-bff6-436f45bf6d4c" containerName="revoke-keys" Oct 31 01:26:01.036310 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.036260 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a86b94ab-470a-4de5-bff6-436f45bf6d4c" containerName="revoke-keys" Oct 31 01:26:01.036310 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.036287 5719 cpu_manager.go:414] "containerMap: removing container" podUID="572aeed2-eff3-404e-bc84-77a6652f5926" containerName="payload-processing" Oct 31 01:26:01.036310 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.036292 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572aeed2-eff3-404e-bc84-77a6652f5926" containerName="payload-processing" Oct 31 01:26:01.036414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.036379 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a86b94ab-470a-4de5-bff6-436f45bf6d4c" containerName="revoke-keys" Oct 31 01:26:01.036414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.036392 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="572aeed2-eff3-404e-bc84-77a6652f5926" containerName="payload-processing" Oct 31 01:26:01.040199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.040182 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.115961 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.115933 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.116108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.115967 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.116108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.116049 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wbj6\" (UniqueName: \"kubernetes.io/projected/e03426c9-0637-4378-ac51-c0718d00965b-kube-api-access-2wbj6\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.128510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.128476 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:26:01.128510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.128486 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:26:01.128510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.128485 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:26:01.128744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.128524 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:26:01.131307 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.131289 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv"] Oct 31 01:26:01.216801 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.216762 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.216801 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.216803 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.216971 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.216839 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wbj6\" (UniqueName: \"kubernetes.io/projected/e03426c9-0637-4378-ac51-c0718d00965b-kube-api-access-2wbj6\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.217472 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.217456 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.219248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.219232 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.276242 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.276219 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wbj6\" (UniqueName: \"kubernetes.io/projected/e03426c9-0637-4378-ac51-c0718d00965b-kube-api-access-2wbj6\") pod \"maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.353452 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.353418 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:01.726862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:01.726797 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv"] Oct 31 01:26:01.732563 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:26:01.732529 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03426c9_0637_4378_ac51_c0718d00965b.slice/crio-a189800fff21b960531be4942a8badaff3fdca0236bed3e6dcdd8a57f70d05d8 WatchSource:0}: Error finding container a189800fff21b960531be4942a8badaff3fdca0236bed3e6dcdd8a57f70d05d8: Status 404 returned error can't find the container with id a189800fff21b960531be4942a8badaff3fdca0236bed3e6dcdd8a57f70d05d8 Oct 31 01:26:02.327046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:02.327005 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" event={"ID":"e03426c9-0637-4378-ac51-c0718d00965b","Type":"ContainerStarted","Data":"c495087998c0e5ea77494d700f6cf127fee8417cc959bf7b9673b4f8a0c144da"} Oct 31 01:26:02.327419 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:02.327062 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" event={"ID":"e03426c9-0637-4378-ac51-c0718d00965b","Type":"ContainerStarted","Data":"a189800fff21b960531be4942a8badaff3fdca0236bed3e6dcdd8a57f70d05d8"} Oct 31 01:26:02.439787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:02.439740 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" podStartSLOduration=2.439726981 podStartE2EDuration="2.439726981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:02.438997058 +0000 UTC m=+3443.975139363" watchObservedRunningTime="2026-10-31 01:26:02.439726981 +0000 UTC m=+3443.975869286" Oct 31 01:26:03.009471 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:03.009386 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:26:03.336867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:03.336837 5719 generic.go:356] "Generic (PLEG): container finished" podID="e03426c9-0637-4378-ac51-c0718d00965b" containerID="c495087998c0e5ea77494d700f6cf127fee8417cc959bf7b9673b4f8a0c144da" exitCode=0 Oct 31 01:26:03.337338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:03.336904 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" event={"ID":"e03426c9-0637-4378-ac51-c0718d00965b","Type":"ContainerDied","Data":"c495087998c0e5ea77494d700f6cf127fee8417cc959bf7b9673b4f8a0c144da"} Oct 31 01:26:03.367684 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:03.367651 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:03.367684 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:03.367665 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:04.414959 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:04.414924 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:04.484356 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.484328 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:04.499938 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.499914 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l"] Oct 31 01:26:04.548109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.548086 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03426c9-0637-4378-ac51-c0718d00965b-kube-api-access-2wbj6\" (UniqueName: \"kubernetes.io/projected/e03426c9-0637-4378-ac51-c0718d00965b-kube-api-access-2wbj6\") pod \"e03426c9-0637-4378-ac51-c0718d00965b\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " Oct 31 01:26:04.548220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.548175 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-xks-ca\") pod \"e03426c9-0637-4378-ac51-c0718d00965b\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " Oct 31 01:26:04.548220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.548212 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-openshift-ca\") pod \"e03426c9-0637-4378-ac51-c0718d00965b\" (UID: \"e03426c9-0637-4378-ac51-c0718d00965b\") " Oct 31 01:26:04.548567 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.548542 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-openshift-ca" pod "e03426c9-0637-4378-ac51-c0718d00965b" (UID: "e03426c9-0637-4378-ac51-c0718d00965b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:04.550205 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.550185 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-xks-ca" pod "e03426c9-0637-4378-ac51-c0718d00965b" (UID: "e03426c9-0637-4378-ac51-c0718d00965b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:26:04.550297 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.550285 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03426c9-0637-4378-ac51-c0718d00965b-kube-api-access-2wbj6" pod "e03426c9-0637-4378-ac51-c0718d00965b" (UID: "e03426c9-0637-4378-ac51-c0718d00965b"). InnerVolumeSpecName "kube-api-access-2wbj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:04.616815 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.616776 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp"] Oct 31 01:26:04.649865 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.649830 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:04.649865 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.649861 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wbj6\" (UniqueName: \"kubernetes.io/projected/e03426c9-0637-4378-ac51-c0718d00965b-kube-api-access-2wbj6\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:04.650033 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:04.649879 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03426c9-0637-4378-ac51-c0718d00965b-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:05.047465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:05.047377 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn"] Oct 31 01:26:05.050550 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:05.050522 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-089afb46-297sn"] Oct 31 01:26:05.351097 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:05.351070 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" Oct 31 01:26:05.351267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:05.351113 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv" event={"ID":"e03426c9-0637-4378-ac51-c0718d00965b","Type":"ContainerDied","Data":"a189800fff21b960531be4942a8badaff3fdca0236bed3e6dcdd8a57f70d05d8"} Oct 31 01:26:05.351267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:05.351160 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a189800fff21b960531be4942a8badaff3fdca0236bed3e6dcdd8a57f70d05d8" Oct 31 01:26:06.281762 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:06.281724 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:06.603597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:06.603569 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8165411-d79e-4415-9f8a-3cfa6ede68d5" path="/var/lib/kubelet/pods/e8165411-d79e-4415-9f8a-3cfa6ede68d5/volumes" Oct 31 01:26:09.501134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.501091 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" podUID="dda90758-30d5-41f7-b7bb-3ffa1af55fe3" containerName="payload-processing" containerID="cri-o://e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e" gracePeriod=25 Oct 31 01:26:09.617230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.617188 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" podUID="22c20645-c2ae-4e73-852b-50932e28eede" containerName="payload-pre-processing" containerID="cri-o://8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9" gracePeriod=25 Oct 31 01:26:09.770017 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.769948 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:26:09.805232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.805207 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-plugins-config-volume\") pod \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\" (UID: \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\") " Oct 31 01:26:09.805366 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.805241 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-kube-api-access-jkqwp\" (UniqueName: \"kubernetes.io/projected/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-kube-api-access-jkqwp\") pod \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\" (UID: \"dda90758-30d5-41f7-b7bb-3ffa1af55fe3\") " Oct 31 01:26:09.805599 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.805576 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-plugins-config-volume" pod "dda90758-30d5-41f7-b7bb-3ffa1af55fe3" (UID: "dda90758-30d5-41f7-b7bb-3ffa1af55fe3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:09.805744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.805729 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:09.807543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.807523 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-kube-api-access-jkqwp" pod "dda90758-30d5-41f7-b7bb-3ffa1af55fe3" (UID: "dda90758-30d5-41f7-b7bb-3ffa1af55fe3"). InnerVolumeSpecName "kube-api-access-jkqwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:09.866382 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.866362 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:26:09.906892 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.906865 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c20645-c2ae-4e73-852b-50932e28eede-kube-api-access-x847c\" (UniqueName: \"kubernetes.io/projected/22c20645-c2ae-4e73-852b-50932e28eede-kube-api-access-x847c\") pod \"22c20645-c2ae-4e73-852b-50932e28eede\" (UID: \"22c20645-c2ae-4e73-852b-50932e28eede\") " Oct 31 01:26:09.907054 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.907037 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c20645-c2ae-4e73-852b-50932e28eede-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c20645-c2ae-4e73-852b-50932e28eede-plugins-config-volume\") pod \"22c20645-c2ae-4e73-852b-50932e28eede\" (UID: \"22c20645-c2ae-4e73-852b-50932e28eede\") " Oct 31 01:26:09.907403 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.907385 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqwp\" (UniqueName: \"kubernetes.io/projected/dda90758-30d5-41f7-b7bb-3ffa1af55fe3-kube-api-access-jkqwp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:09.907674 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.907430 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c20645-c2ae-4e73-852b-50932e28eede-plugins-config-volume" pod "22c20645-c2ae-4e73-852b-50932e28eede" (UID: "22c20645-c2ae-4e73-852b-50932e28eede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:09.909124 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:09.909102 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c20645-c2ae-4e73-852b-50932e28eede-kube-api-access-x847c" pod "22c20645-c2ae-4e73-852b-50932e28eede" (UID: "22c20645-c2ae-4e73-852b-50932e28eede"). InnerVolumeSpecName "kube-api-access-x847c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:10.009209 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.009171 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x847c\" (UniqueName: \"kubernetes.io/projected/22c20645-c2ae-4e73-852b-50932e28eede-kube-api-access-x847c\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:10.009209 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.009209 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c20645-c2ae-4e73-852b-50932e28eede-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:10.383907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.383874 5719 generic.go:356] "Generic (PLEG): container finished" podID="dda90758-30d5-41f7-b7bb-3ffa1af55fe3" containerID="e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e" exitCode=0 Oct 31 01:26:10.384117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.383928 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" Oct 31 01:26:10.384117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.383960 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" event={"ID":"dda90758-30d5-41f7-b7bb-3ffa1af55fe3","Type":"ContainerDied","Data":"e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e"} Oct 31 01:26:10.384117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.384007 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l" event={"ID":"dda90758-30d5-41f7-b7bb-3ffa1af55fe3","Type":"ContainerDied","Data":"455e04694c3bfef70a1af212c70b3613244fe6df701911b27ffa2f67f1468c2e"} Oct 31 01:26:10.384117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.384018 5719 scope.go:122] "RemoveContainer" containerID="e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e" Oct 31 01:26:10.385567 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.385544 5719 generic.go:356] "Generic (PLEG): container finished" podID="22c20645-c2ae-4e73-852b-50932e28eede" containerID="8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9" exitCode=0 Oct 31 01:26:10.385715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.385583 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" Oct 31 01:26:10.385715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.385635 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" event={"ID":"22c20645-c2ae-4e73-852b-50932e28eede","Type":"ContainerDied","Data":"8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9"} Oct 31 01:26:10.385845 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.385723 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp" event={"ID":"22c20645-c2ae-4e73-852b-50932e28eede","Type":"ContainerDied","Data":"036226c19dd209ea3e8208a6fc549c0d74bec9a4e2da30090d9bddf190a6c449"} Oct 31 01:26:10.399039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.399019 5719 scope.go:122] "RemoveContainer" containerID="e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e" Oct 31 01:26:10.399319 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:10.399301 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e\": container with ID starting with e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e not found: ID does not exist" containerID="e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e" Oct 31 01:26:10.399368 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.399326 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e"} err="failed to get container status \"e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e\": rpc error: code = NotFound desc = could not find container \"e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e\": container with ID starting with e217611d787663e806f5c3cf7cd92a3450af9138a96a68fec50bda0934ee193e not found: ID does not exist" Oct 31 01:26:10.399368 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.399344 5719 scope.go:122] "RemoveContainer" containerID="8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9" Oct 31 01:26:10.408748 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.408732 5719 scope.go:122] "RemoveContainer" containerID="8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9" Oct 31 01:26:10.409002 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:10.408986 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9\": container with ID starting with 8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9 not found: ID does not exist" containerID="8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9" Oct 31 01:26:10.409068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.409008 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9"} err="failed to get container status \"8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9\": rpc error: code = NotFound desc = could not find container \"8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9\": container with ID starting with 8e230c97d4d1cb346a31508fa679366a5aad9044d520e786247fd0bde2cbf4a9 not found: ID does not exist" Oct 31 01:26:10.543730 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.543703 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp"] Oct 31 01:26:10.591577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.591554 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-08fcc4-7698dffc5-lkxlp"] Oct 31 01:26:10.604044 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.604024 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c20645-c2ae-4e73-852b-50932e28eede" path="/var/lib/kubelet/pods/22c20645-c2ae-4e73-852b-50932e28eede/volumes" Oct 31 01:26:10.699325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.699276 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l"] Oct 31 01:26:10.703375 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:10.703351 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-08fcc4-7fc46bc755-qcr2l"] Oct 31 01:26:12.603207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:12.603174 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda90758-30d5-41f7-b7bb-3ffa1af55fe3" path="/var/lib/kubelet/pods/dda90758-30d5-41f7-b7bb-3ffa1af55fe3/volumes" Oct 31 01:26:14.485682 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:14.485651 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:21.348607 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:21.348572 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:24.502366 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:24.502327 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:27.059724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:27.059673 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24"] Oct 31 01:26:27.107300 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:27.107275 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ce3765-tqg24"] Oct 31 01:26:28.604709 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:28.604659 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d05cbf7-43e9-4d26-a7b1-fd0b988d8533" path="/var/lib/kubelet/pods/1d05cbf7-43e9-4d26-a7b1-fd0b988d8533/volumes" Oct 31 01:26:30.057484 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:30.057437 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv"] Oct 31 01:26:30.104807 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:30.104776 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4287e6ff-9vdkv"] Oct 31 01:26:30.602985 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:30.602956 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84e9dfe-0fc6-4e51-98e2-bf49beeace1a" path="/var/lib/kubelet/pods/c84e9dfe-0fc6-4e51-98e2-bf49beeace1a/volumes" Oct 31 01:26:31.167432 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:31.167391 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx"] Oct 31 01:26:31.214459 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:31.214432 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3aca3f-9gqwx"] Oct 31 01:26:32.602616 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:32.602585 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460fa0d9-d41c-4dcc-a62f-89c1697a28d5" path="/var/lib/kubelet/pods/460fa0d9-d41c-4dcc-a62f-89c1697a28d5/volumes" Oct 31 01:26:33.010738 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:33.010629 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:26:34.569992 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:34.569960 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:36.339489 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:26:36.339446 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257ddf2a_6ad0_4c0e_ac2b_2fb304f8256a.slice/crio-conmon-5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:26:38.048088 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:38.048045 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk"] Oct 31 01:26:38.095260 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:38.095232 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a067529-b4gzk"] Oct 31 01:26:38.603377 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:38.603346 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7538c012-87e1-4171-88a8-1e7375d6de48" path="/var/lib/kubelet/pods/7538c012-87e1-4171-88a8-1e7375d6de48/volumes" Oct 31 01:26:40.514249 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514216 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p"] Oct 31 01:26:40.514752 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514737 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e03426c9-0637-4378-ac51-c0718d00965b" containerName="revoke-keys" Oct 31 01:26:40.514752 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514753 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03426c9-0637-4378-ac51-c0718d00965b" containerName="revoke-keys" Oct 31 01:26:40.514857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514772 5719 cpu_manager.go:414] "containerMap: removing container" podUID="dda90758-30d5-41f7-b7bb-3ffa1af55fe3" containerName="payload-processing" Oct 31 01:26:40.514857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514777 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda90758-30d5-41f7-b7bb-3ffa1af55fe3" containerName="payload-processing" Oct 31 01:26:40.514857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514796 5719 cpu_manager.go:414] "containerMap: removing container" podUID="22c20645-c2ae-4e73-852b-50932e28eede" containerName="payload-pre-processing" Oct 31 01:26:40.514857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514801 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c20645-c2ae-4e73-852b-50932e28eede" containerName="payload-pre-processing" Oct 31 01:26:40.514988 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514894 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03426c9-0637-4378-ac51-c0718d00965b" containerName="revoke-keys" Oct 31 01:26:40.514988 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514902 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda90758-30d5-41f7-b7bb-3ffa1af55fe3" containerName="payload-processing" Oct 31 01:26:40.514988 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.514913 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c20645-c2ae-4e73-852b-50932e28eede" containerName="payload-pre-processing" Oct 31 01:26:40.518289 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.518272 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.570885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.570857 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:26:40.571007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.570908 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:26:40.591004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.590973 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0a0594-3186-487f-a948-92519910593c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.591169 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.591149 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fppgw\" (UniqueName: \"kubernetes.io/projected/9e0a0594-3186-487f-a948-92519910593c-kube-api-access-fppgw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.591330 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.591307 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a0594-3186-487f-a948-92519910593c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.618645 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.618618 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:26:40.618800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.618744 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:26:40.621271 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.621253 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p"] Oct 31 01:26:40.692708 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.692673 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fppgw\" (UniqueName: \"kubernetes.io/projected/9e0a0594-3186-487f-a948-92519910593c-kube-api-access-fppgw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.692828 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.692782 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a0594-3186-487f-a948-92519910593c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.692828 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.692823 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0a0594-3186-487f-a948-92519910593c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.693362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.693342 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a0594-3186-487f-a948-92519910593c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.695184 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.695163 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0a0594-3186-487f-a948-92519910593c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.751965 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.751939 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fppgw\" (UniqueName: \"kubernetes.io/projected/9e0a0594-3186-487f-a948-92519910593c-kube-api-access-fppgw\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:40.831435 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:40.831415 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:41.047309 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:41.047274 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl"] Oct 31 01:26:41.050633 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:41.050604 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a209c12-9r6gl"] Oct 31 01:26:41.207296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:41.207216 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p"] Oct 31 01:26:41.213668 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:26:41.213616 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e0a0594_3186_487f_a948_92519910593c.slice/crio-90c181b7b2ad18c62239de0607e33bc24117c664ab8e196a16e93ef94f2ddfd8 WatchSource:0}: Error finding container 90c181b7b2ad18c62239de0607e33bc24117c664ab8e196a16e93ef94f2ddfd8: Status 404 returned error can't find the container with id 90c181b7b2ad18c62239de0607e33bc24117c664ab8e196a16e93ef94f2ddfd8 Oct 31 01:26:41.588627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:41.588596 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" event={"ID":"9e0a0594-3186-487f-a948-92519910593c","Type":"ContainerStarted","Data":"7e3fea8caca6c69ef8cc96dc97fac77183f6124a2d9bc6ab60aad59cbab67ab5"} Oct 31 01:26:41.589011 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:41.588660 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" event={"ID":"9e0a0594-3186-487f-a948-92519910593c","Type":"ContainerStarted","Data":"90c181b7b2ad18c62239de0607e33bc24117c664ab8e196a16e93ef94f2ddfd8"} Oct 31 01:26:41.703192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:41.703151 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" podStartSLOduration=1.7031391390000001 podStartE2EDuration="1.703139139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:41.701093159 +0000 UTC m=+3483.237235465" watchObservedRunningTime="2026-10-31 01:26:41.703139139 +0000 UTC m=+3483.239281486" Oct 31 01:26:42.598760 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:42.598731 5719 generic.go:356] "Generic (PLEG): container finished" podID="9e0a0594-3186-487f-a948-92519910593c" containerID="7e3fea8caca6c69ef8cc96dc97fac77183f6124a2d9bc6ab60aad59cbab67ab5" exitCode=0 Oct 31 01:26:42.603533 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:42.603507 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47db979-411d-4ed8-8e33-77fe72d1e63e" path="/var/lib/kubelet/pods/a47db979-411d-4ed8-8e33-77fe72d1e63e/volumes" Oct 31 01:26:42.603923 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:42.603903 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" event={"ID":"9e0a0594-3186-487f-a948-92519910593c","Type":"ContainerDied","Data":"7e3fea8caca6c69ef8cc96dc97fac77183f6124a2d9bc6ab60aad59cbab67ab5"} Oct 31 01:26:43.745818 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.745790 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:43.826370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.826349 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0a0594-3186-487f-a948-92519910593c-kube-api-access-fppgw\" (UniqueName: \"kubernetes.io/projected/9e0a0594-3186-487f-a948-92519910593c-kube-api-access-fppgw\") pod \"9e0a0594-3186-487f-a948-92519910593c\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " Oct 31 01:26:43.826510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.826381 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0a0594-3186-487f-a948-92519910593c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a0594-3186-487f-a948-92519910593c-maas-api-openshift-ca\") pod \"9e0a0594-3186-487f-a948-92519910593c\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " Oct 31 01:26:43.826510 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.826410 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0a0594-3186-487f-a948-92519910593c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0a0594-3186-487f-a948-92519910593c-maas-api-xks-ca\") pod \"9e0a0594-3186-487f-a948-92519910593c\" (UID: \"9e0a0594-3186-487f-a948-92519910593c\") " Oct 31 01:26:43.826772 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.826749 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0a0594-3186-487f-a948-92519910593c-maas-api-openshift-ca" pod "9e0a0594-3186-487f-a948-92519910593c" (UID: "9e0a0594-3186-487f-a948-92519910593c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:43.828548 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.828529 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0a0594-3186-487f-a948-92519910593c-maas-api-xks-ca" pod "9e0a0594-3186-487f-a948-92519910593c" (UID: "9e0a0594-3186-487f-a948-92519910593c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:26:43.828606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.828555 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0a0594-3186-487f-a948-92519910593c-kube-api-access-fppgw" pod "9e0a0594-3186-487f-a948-92519910593c" (UID: "9e0a0594-3186-487f-a948-92519910593c"). InnerVolumeSpecName "kube-api-access-fppgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:43.927655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.927591 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fppgw\" (UniqueName: \"kubernetes.io/projected/9e0a0594-3186-487f-a948-92519910593c-kube-api-access-fppgw\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.927655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.927617 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a0594-3186-487f-a948-92519910593c-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:43.927655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:43.927628 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e0a0594-3186-487f-a948-92519910593c-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:26:44.374360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:44.374328 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:26:44.614405 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:44.614373 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" Oct 31 01:26:44.614555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:44.614404 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p" event={"ID":"9e0a0594-3186-487f-a948-92519910593c","Type":"ContainerDied","Data":"90c181b7b2ad18c62239de0607e33bc24117c664ab8e196a16e93ef94f2ddfd8"} Oct 31 01:26:44.614555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:44.614444 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90c181b7b2ad18c62239de0607e33bc24117c664ab8e196a16e93ef94f2ddfd8" Oct 31 01:26:52.045936 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:52.045902 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7"] Oct 31 01:26:52.154573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:52.154535 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv"] Oct 31 01:26:57.046643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:57.046538 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" podUID="364f5aab-0dc0-4eef-a6a2-0c3966206372" containerName="payload-processing" containerID="cri-o://18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d" gracePeriod=25 Oct 31 01:26:57.155038 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:57.155004 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" podUID="839c8447-8e55-481f-ac4e-2363cfe5792b" containerName="payload-pre-processing" containerID="cri-o://23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b" gracePeriod=25 Oct 31 01:26:57.571527 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:57.571495 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" podUID="839c8447-8e55-481f-ac4e-2363cfe5792b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:26:57.571676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:26:57.571508 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" podUID="364f5aab-0dc0-4eef-a6a2-0c3966206372" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:27:03.010249 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:27:03.010214 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:27:07.571992 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:07.571959 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" podUID="364f5aab-0dc0-4eef-a6a2-0c3966206372" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:27:07.572455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:07.571970 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" podUID="839c8447-8e55-481f-ac4e-2363cfe5792b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:27:11.335848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:11.335816 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v"] Oct 31 01:27:11.383172 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:11.383145 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d067bd-bxh2v"] Oct 31 01:27:12.603878 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:12.603847 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0001331a-9326-4238-8893-9548aa95f208" path="/var/lib/kubelet/pods/0001331a-9326-4238-8893-9548aa95f208/volumes" Oct 31 01:27:12.677643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:12.677615 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls"] Oct 31 01:27:12.724573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:12.724551 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6b5091-f99ls"] Oct 31 01:27:14.602838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:14.602809 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a" path="/var/lib/kubelet/pods/257ddf2a-6ad0-4c0e-ac2b-2fb304f8256a/volumes" Oct 31 01:27:17.316440 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.316414 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:27:17.352956 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.352925 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364f5aab-0dc0-4eef-a6a2-0c3966206372-kube-api-access-ls6pd\" (UniqueName: \"kubernetes.io/projected/364f5aab-0dc0-4eef-a6a2-0c3966206372-kube-api-access-ls6pd\") pod \"364f5aab-0dc0-4eef-a6a2-0c3966206372\" (UID: \"364f5aab-0dc0-4eef-a6a2-0c3966206372\") " Oct 31 01:27:17.353134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.353011 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364f5aab-0dc0-4eef-a6a2-0c3966206372-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364f5aab-0dc0-4eef-a6a2-0c3966206372-plugins-config-volume\") pod \"364f5aab-0dc0-4eef-a6a2-0c3966206372\" (UID: \"364f5aab-0dc0-4eef-a6a2-0c3966206372\") " Oct 31 01:27:17.353871 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.353841 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364f5aab-0dc0-4eef-a6a2-0c3966206372-plugins-config-volume" pod "364f5aab-0dc0-4eef-a6a2-0c3966206372" (UID: "364f5aab-0dc0-4eef-a6a2-0c3966206372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:17.355768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.355739 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364f5aab-0dc0-4eef-a6a2-0c3966206372-kube-api-access-ls6pd" pod "364f5aab-0dc0-4eef-a6a2-0c3966206372" (UID: "364f5aab-0dc0-4eef-a6a2-0c3966206372"). InnerVolumeSpecName "kube-api-access-ls6pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:17.394268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.394245 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:27:17.454249 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.454218 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839c8447-8e55-481f-ac4e-2363cfe5792b-kube-api-access-6fqlf\" (UniqueName: \"kubernetes.io/projected/839c8447-8e55-481f-ac4e-2363cfe5792b-kube-api-access-6fqlf\") pod \"839c8447-8e55-481f-ac4e-2363cfe5792b\" (UID: \"839c8447-8e55-481f-ac4e-2363cfe5792b\") " Oct 31 01:27:17.454401 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.454304 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839c8447-8e55-481f-ac4e-2363cfe5792b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c8447-8e55-481f-ac4e-2363cfe5792b-plugins-config-volume\") pod \"839c8447-8e55-481f-ac4e-2363cfe5792b\" (UID: \"839c8447-8e55-481f-ac4e-2363cfe5792b\") " Oct 31 01:27:17.454623 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.454605 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls6pd\" (UniqueName: \"kubernetes.io/projected/364f5aab-0dc0-4eef-a6a2-0c3966206372-kube-api-access-ls6pd\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:17.454754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.454625 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/364f5aab-0dc0-4eef-a6a2-0c3966206372-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:17.454754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.454639 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839c8447-8e55-481f-ac4e-2363cfe5792b-plugins-config-volume" pod "839c8447-8e55-481f-ac4e-2363cfe5792b" (UID: "839c8447-8e55-481f-ac4e-2363cfe5792b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:17.456552 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.456528 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839c8447-8e55-481f-ac4e-2363cfe5792b-kube-api-access-6fqlf" pod "839c8447-8e55-481f-ac4e-2363cfe5792b" (UID: "839c8447-8e55-481f-ac4e-2363cfe5792b"). InnerVolumeSpecName "kube-api-access-6fqlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:17.555722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.555671 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fqlf\" (UniqueName: \"kubernetes.io/projected/839c8447-8e55-481f-ac4e-2363cfe5792b-kube-api-access-6fqlf\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:17.555722 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.555718 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c8447-8e55-481f-ac4e-2363cfe5792b-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:17.816365 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.816336 5719 generic.go:356] "Generic (PLEG): container finished" podID="839c8447-8e55-481f-ac4e-2363cfe5792b" containerID="23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b" exitCode=0 Oct 31 01:27:17.816612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.816386 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" Oct 31 01:27:17.816612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.816428 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" event={"ID":"839c8447-8e55-481f-ac4e-2363cfe5792b","Type":"ContainerDied","Data":"23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b"} Oct 31 01:27:17.816612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.816497 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv" event={"ID":"839c8447-8e55-481f-ac4e-2363cfe5792b","Type":"ContainerDied","Data":"996953a77bad19773601bc5564d67e7e874c0c2132e95b3dd2b97c17e52800f8"} Oct 31 01:27:17.816612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.816528 5719 scope.go:122] "RemoveContainer" containerID="23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b" Oct 31 01:27:17.817788 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.817767 5719 generic.go:356] "Generic (PLEG): container finished" podID="364f5aab-0dc0-4eef-a6a2-0c3966206372" containerID="18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d" exitCode=0 Oct 31 01:27:17.817864 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.817815 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" Oct 31 01:27:17.817864 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.817845 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" event={"ID":"364f5aab-0dc0-4eef-a6a2-0c3966206372","Type":"ContainerDied","Data":"18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d"} Oct 31 01:27:17.817946 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.817887 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7" event={"ID":"364f5aab-0dc0-4eef-a6a2-0c3966206372","Type":"ContainerDied","Data":"640ebd7f049cbea309306d39c80418c2d01c9a0ce9dfd54c5a5f90fb03d5fe08"} Oct 31 01:27:17.826584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.826561 5719 scope.go:122] "RemoveContainer" containerID="23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b" Oct 31 01:27:17.826890 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:27:17.826871 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b\": container with ID starting with 23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b not found: ID does not exist" containerID="23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b" Oct 31 01:27:17.826980 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.826896 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b"} err="failed to get container status \"23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b\": rpc error: code = NotFound desc = could not find container \"23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b\": container with ID starting with 23e0bc8526fdbd4f601371f387f4c97c2abf26da582a88eccb0cf3f923fe2a2b not found: ID does not exist" Oct 31 01:27:17.826980 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.826912 5719 scope.go:122] "RemoveContainer" containerID="18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d" Oct 31 01:27:17.835540 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.835525 5719 scope.go:122] "RemoveContainer" containerID="18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d" Oct 31 01:27:17.835815 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:27:17.835795 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d\": container with ID starting with 18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d not found: ID does not exist" containerID="18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d" Oct 31 01:27:17.835923 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.835819 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d"} err="failed to get container status \"18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d\": rpc error: code = NotFound desc = could not find container \"18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d\": container with ID starting with 18407cda1762aa3750e9350c493943eb6839121abd5733f51c48ed8477c2064d not found: ID does not exist" Oct 31 01:27:17.977374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:17.977348 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7"] Oct 31 01:27:18.025083 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:18.025053 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ba34eb-6ddd8d9c5c-jw2x7"] Oct 31 01:27:18.087056 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:18.086999 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv"] Oct 31 01:27:18.092932 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:18.092910 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ba34eb-df7485b87-rc2mv"] Oct 31 01:27:18.603280 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:18.603251 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364f5aab-0dc0-4eef-a6a2-0c3966206372" path="/var/lib/kubelet/pods/364f5aab-0dc0-4eef-a6a2-0c3966206372/volumes" Oct 31 01:27:18.603863 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:18.603843 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839c8447-8e55-481f-ac4e-2363cfe5792b" path="/var/lib/kubelet/pods/839c8447-8e55-481f-ac4e-2363cfe5792b/volumes" Oct 31 01:27:21.233128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233095 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p"] Oct 31 01:27:21.233619 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233604 5719 cpu_manager.go:414] "containerMap: removing container" podUID="364f5aab-0dc0-4eef-a6a2-0c3966206372" containerName="payload-processing" Oct 31 01:27:21.233669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233620 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364f5aab-0dc0-4eef-a6a2-0c3966206372" containerName="payload-processing" Oct 31 01:27:21.233669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233638 5719 cpu_manager.go:414] "containerMap: removing container" podUID="839c8447-8e55-481f-ac4e-2363cfe5792b" containerName="payload-pre-processing" Oct 31 01:27:21.233669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233643 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839c8447-8e55-481f-ac4e-2363cfe5792b" containerName="payload-pre-processing" Oct 31 01:27:21.233669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233654 5719 cpu_manager.go:414] "containerMap: removing container" podUID="9e0a0594-3186-487f-a948-92519910593c" containerName="revoke-keys" Oct 31 01:27:21.233669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233658 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0a0594-3186-487f-a948-92519910593c" containerName="revoke-keys" Oct 31 01:27:21.233861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233777 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="364f5aab-0dc0-4eef-a6a2-0c3966206372" containerName="payload-processing" Oct 31 01:27:21.233861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233788 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="839c8447-8e55-481f-ac4e-2363cfe5792b" containerName="payload-pre-processing" Oct 31 01:27:21.233861 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.233796 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0a0594-3186-487f-a948-92519910593c" containerName="revoke-keys" Oct 31 01:27:21.238893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.238873 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.295112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.295085 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7h76\" (UniqueName: \"kubernetes.io/projected/0e66432f-688b-4aa6-b770-6391ab662009-kube-api-access-k7h76\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.295276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.295118 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.295276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.295168 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.337081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.337034 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:27:21.337081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.337056 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:27:21.337081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.337033 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:27:21.337389 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.337164 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:27:21.340055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.340030 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p"] Oct 31 01:27:21.396247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.396222 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7h76\" (UniqueName: \"kubernetes.io/projected/0e66432f-688b-4aa6-b770-6391ab662009-kube-api-access-k7h76\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.396378 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.396255 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.396378 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.396297 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.397057 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.397035 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.398927 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.398907 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.455471 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.455449 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7h76\" (UniqueName: \"kubernetes.io/projected/0e66432f-688b-4aa6-b770-6391ab662009-kube-api-access-k7h76\") pod \"maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.553089 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.553038 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:21.927846 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:21.927802 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p"] Oct 31 01:27:21.939771 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:27:21.934485 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e66432f_688b_4aa6_b770_6391ab662009.slice/crio-ed1a4e569d1f8dd931b77f1cfbc0642cf644c0633939b3970eb2a2bc6988e0b1 WatchSource:0}: Error finding container ed1a4e569d1f8dd931b77f1cfbc0642cf644c0633939b3970eb2a2bc6988e0b1: Status 404 returned error can't find the container with id ed1a4e569d1f8dd931b77f1cfbc0642cf644c0633939b3970eb2a2bc6988e0b1 Oct 31 01:27:22.856951 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:22.856922 5719 generic.go:356] "Generic (PLEG): container finished" podID="0e66432f-688b-4aa6-b770-6391ab662009" containerID="9811bc9e19f5f8a80d1d1b184f4fbf7f13be13a496184e7cd6cb65d80164f528" exitCode=0 Oct 31 01:27:22.857352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:22.856980 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" event={"ID":"0e66432f-688b-4aa6-b770-6391ab662009","Type":"ContainerDied","Data":"9811bc9e19f5f8a80d1d1b184f4fbf7f13be13a496184e7cd6cb65d80164f528"} Oct 31 01:27:22.857352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:22.857059 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" event={"ID":"0e66432f-688b-4aa6-b770-6391ab662009","Type":"ContainerStarted","Data":"ed1a4e569d1f8dd931b77f1cfbc0642cf644c0633939b3970eb2a2bc6988e0b1"} Oct 31 01:27:23.996853 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:23.996821 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv"] Oct 31 01:27:24.006519 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.006497 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:24.042754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.042728 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-eb68dd-mk5pv"] Oct 31 01:27:24.127072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.126992 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-openshift-ca\") pod \"0e66432f-688b-4aa6-b770-6391ab662009\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " Oct 31 01:27:24.127240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.127094 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e66432f-688b-4aa6-b770-6391ab662009-kube-api-access-k7h76\" (UniqueName: \"kubernetes.io/projected/0e66432f-688b-4aa6-b770-6391ab662009-kube-api-access-k7h76\") pod \"0e66432f-688b-4aa6-b770-6391ab662009\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " Oct 31 01:27:24.127240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.127125 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-xks-ca\") pod \"0e66432f-688b-4aa6-b770-6391ab662009\" (UID: \"0e66432f-688b-4aa6-b770-6391ab662009\") " Oct 31 01:27:24.127430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.127404 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-openshift-ca" pod "0e66432f-688b-4aa6-b770-6391ab662009" (UID: "0e66432f-688b-4aa6-b770-6391ab662009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:24.129583 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.129556 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e66432f-688b-4aa6-b770-6391ab662009-kube-api-access-k7h76" pod "0e66432f-688b-4aa6-b770-6391ab662009" (UID: "0e66432f-688b-4aa6-b770-6391ab662009"). InnerVolumeSpecName "kube-api-access-k7h76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:24.130767 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.129927 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-xks-ca" pod "0e66432f-688b-4aa6-b770-6391ab662009" (UID: "0e66432f-688b-4aa6-b770-6391ab662009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:24.228401 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.228361 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:24.228401 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.228393 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7h76\" (UniqueName: \"kubernetes.io/projected/0e66432f-688b-4aa6-b770-6391ab662009-kube-api-access-k7h76\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:24.228401 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.228408 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e66432f-688b-4aa6-b770-6391ab662009-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:24.604020 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.603989 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03426c9-0637-4378-ac51-c0718d00965b" path="/var/lib/kubelet/pods/e03426c9-0637-4378-ac51-c0718d00965b/volumes" Oct 31 01:27:24.872134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.872050 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" Oct 31 01:27:24.872286 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.872054 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p" event={"ID":"0e66432f-688b-4aa6-b770-6391ab662009","Type":"ContainerDied","Data":"ed1a4e569d1f8dd931b77f1cfbc0642cf644c0633939b3970eb2a2bc6988e0b1"} Oct 31 01:27:24.872286 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:24.872222 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed1a4e569d1f8dd931b77f1cfbc0642cf644c0633939b3970eb2a2bc6988e0b1" Oct 31 01:27:25.368007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:25.367976 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb"] Oct 31 01:27:25.370669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:25.370647 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-08fcc4-p9nfb"] Oct 31 01:27:25.728317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:25.728239 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9"] Oct 31 01:27:26.606512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:26.606470 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a86b94ab-470a-4de5-bff6-436f45bf6d4c" path="/var/lib/kubelet/pods/a86b94ab-470a-4de5-bff6-436f45bf6d4c/volumes" Oct 31 01:27:33.006662 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.006590 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx"] Oct 31 01:27:33.007188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.007172 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0e66432f-688b-4aa6-b770-6391ab662009" containerName="revoke-keys" Oct 31 01:27:33.007227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.007190 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e66432f-688b-4aa6-b770-6391ab662009" containerName="revoke-keys" Oct 31 01:27:33.007325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.007314 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e66432f-688b-4aa6-b770-6391ab662009" containerName="revoke-keys" Oct 31 01:27:33.010832 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:27:33.010808 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:27:33.012422 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.012409 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.065656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.065629 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx"] Oct 31 01:27:33.110670 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.110644 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:27:33.110670 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.110659 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:27:33.110873 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.110678 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:27:33.110873 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.110734 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:27:33.126335 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.126309 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwn74\" (UniqueName: \"kubernetes.io/projected/011e3744-fcb9-4729-8882-c8c42797b6dd-kube-api-access-wwn74\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.126560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.126537 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.126767 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.126741 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.227893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.227859 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.228049 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.227999 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwn74\" (UniqueName: \"kubernetes.io/projected/011e3744-fcb9-4729-8882-c8c42797b6dd-kube-api-access-wwn74\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.228049 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.228024 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.228603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.228582 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.230284 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.230261 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.287905 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.287826 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwn74\" (UniqueName: \"kubernetes.io/projected/011e3744-fcb9-4729-8882-c8c42797b6dd-kube-api-access-wwn74\") pod \"maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.326351 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.326322 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:33.701090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.701059 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx"] Oct 31 01:27:33.707822 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:27:33.707796 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011e3744_fcb9_4729_8882_c8c42797b6dd.slice/crio-87e0c256d93aec0dfc9ab3945f9959820c76646813b634017f12b76abc901dc6 WatchSource:0}: Error finding container 87e0c256d93aec0dfc9ab3945f9959820c76646813b634017f12b76abc901dc6: Status 404 returned error can't find the container with id 87e0c256d93aec0dfc9ab3945f9959820c76646813b634017f12b76abc901dc6 Oct 31 01:27:33.933264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:33.933240 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" event={"ID":"011e3744-fcb9-4729-8882-c8c42797b6dd","Type":"ContainerStarted","Data":"87e0c256d93aec0dfc9ab3945f9959820c76646813b634017f12b76abc901dc6"} Oct 31 01:27:34.941013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:34.940982 5719 generic.go:356] "Generic (PLEG): container finished" podID="011e3744-fcb9-4729-8882-c8c42797b6dd" containerID="20c05895309444bda393ba57b6f2d0b3942f9410975e0fdfc7c0aeaf7dbb3393" exitCode=0 Oct 31 01:27:34.941412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:34.941061 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" event={"ID":"011e3744-fcb9-4729-8882-c8c42797b6dd","Type":"ContainerDied","Data":"20c05895309444bda393ba57b6f2d0b3942f9410975e0fdfc7c0aeaf7dbb3393"} Oct 31 01:27:35.499253 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.499218 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl"] Oct 31 01:27:35.503255 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.503231 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.571957 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.571928 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl"] Oct 31 01:27:35.652580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.652549 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.652780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.652588 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.652780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.652744 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddncp\" (UniqueName: \"kubernetes.io/projected/d85b2bf8-32bc-4338-9e3e-18dfa657f136-kube-api-access-ddncp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.754120 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.754044 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddncp\" (UniqueName: \"kubernetes.io/projected/d85b2bf8-32bc-4338-9e3e-18dfa657f136-kube-api-access-ddncp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.754294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.754165 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.754294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.754189 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.754952 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.754930 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.756792 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.756769 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.812914 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.812891 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddncp\" (UniqueName: \"kubernetes.io/projected/d85b2bf8-32bc-4338-9e3e-18dfa657f136-kube-api-access-ddncp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:35.816742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:35.816717 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:36.079331 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.079311 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:36.159301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.159275 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-openshift-ca\") pod \"011e3744-fcb9-4729-8882-c8c42797b6dd\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " Oct 31 01:27:36.159430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.159325 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-xks-ca\") pod \"011e3744-fcb9-4729-8882-c8c42797b6dd\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " Oct 31 01:27:36.159508 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.159455 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011e3744-fcb9-4729-8882-c8c42797b6dd-kube-api-access-wwn74\" (UniqueName: \"kubernetes.io/projected/011e3744-fcb9-4729-8882-c8c42797b6dd-kube-api-access-wwn74\") pod \"011e3744-fcb9-4729-8882-c8c42797b6dd\" (UID: \"011e3744-fcb9-4729-8882-c8c42797b6dd\") " Oct 31 01:27:36.159643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.159618 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-openshift-ca" pod "011e3744-fcb9-4729-8882-c8c42797b6dd" (UID: "011e3744-fcb9-4729-8882-c8c42797b6dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:36.159869 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.159852 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:36.161466 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.161444 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-xks-ca" pod "011e3744-fcb9-4729-8882-c8c42797b6dd" (UID: "011e3744-fcb9-4729-8882-c8c42797b6dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:36.161600 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.161581 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011e3744-fcb9-4729-8882-c8c42797b6dd-kube-api-access-wwn74" pod "011e3744-fcb9-4729-8882-c8c42797b6dd" (UID: "011e3744-fcb9-4729-8882-c8c42797b6dd"). InnerVolumeSpecName "kube-api-access-wwn74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:36.192503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.192477 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl"] Oct 31 01:27:36.199140 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:27:36.199114 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85b2bf8_32bc_4338_9e3e_18dfa657f136.slice/crio-634670f23b39b4bd9d65b833553ef52b3c88186f035310db51a8cacd38bff436 WatchSource:0}: Error finding container 634670f23b39b4bd9d65b833553ef52b3c88186f035310db51a8cacd38bff436: Status 404 returned error can't find the container with id 634670f23b39b4bd9d65b833553ef52b3c88186f035310db51a8cacd38bff436 Oct 31 01:27:36.261098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.261078 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/011e3744-fcb9-4729-8882-c8c42797b6dd-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:36.261098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.261099 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwn74\" (UniqueName: \"kubernetes.io/projected/011e3744-fcb9-4729-8882-c8c42797b6dd-kube-api-access-wwn74\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:36.957125 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.957092 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" event={"ID":"d85b2bf8-32bc-4338-9e3e-18dfa657f136","Type":"ContainerStarted","Data":"6acbdf8c839b14f17a9b2992ddd1a4e6ef9b2b3628b17f2587caddc6a5e7248f"} Oct 31 01:27:36.957301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.957151 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" event={"ID":"d85b2bf8-32bc-4338-9e3e-18dfa657f136","Type":"ContainerStarted","Data":"634670f23b39b4bd9d65b833553ef52b3c88186f035310db51a8cacd38bff436"} Oct 31 01:27:36.958546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.958518 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" event={"ID":"011e3744-fcb9-4729-8882-c8c42797b6dd","Type":"ContainerDied","Data":"87e0c256d93aec0dfc9ab3945f9959820c76646813b634017f12b76abc901dc6"} Oct 31 01:27:36.958546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.958540 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx" Oct 31 01:27:36.958756 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:36.958567 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e0c256d93aec0dfc9ab3945f9959820c76646813b634017f12b76abc901dc6" Oct 31 01:27:37.069079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:37.069035 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" podStartSLOduration=2.069021148 podStartE2EDuration="2.069021148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:37.067872471 +0000 UTC m=+3538.604014790" watchObservedRunningTime="2026-10-31 01:27:37.069021148 +0000 UTC m=+3538.605163452" Oct 31 01:27:37.967372 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:37.967330 5719 generic.go:356] "Generic (PLEG): container finished" podID="d85b2bf8-32bc-4338-9e3e-18dfa657f136" containerID="6acbdf8c839b14f17a9b2992ddd1a4e6ef9b2b3628b17f2587caddc6a5e7248f" exitCode=0 Oct 31 01:27:37.967800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:37.967386 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" event={"ID":"d85b2bf8-32bc-4338-9e3e-18dfa657f136","Type":"ContainerDied","Data":"6acbdf8c839b14f17a9b2992ddd1a4e6ef9b2b3628b17f2587caddc6a5e7248f"} Oct 31 01:27:38.105632 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:38.105595 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:27:39.119779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.119750 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:39.293576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.293510 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-xks-ca\") pod \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " Oct 31 01:27:39.293576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.293546 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85b2bf8-32bc-4338-9e3e-18dfa657f136-kube-api-access-ddncp\" (UniqueName: \"kubernetes.io/projected/d85b2bf8-32bc-4338-9e3e-18dfa657f136-kube-api-access-ddncp\") pod \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " Oct 31 01:27:39.293843 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.293594 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-openshift-ca\") pod \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\" (UID: \"d85b2bf8-32bc-4338-9e3e-18dfa657f136\") " Oct 31 01:27:39.293999 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.293976 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-openshift-ca" pod "d85b2bf8-32bc-4338-9e3e-18dfa657f136" (UID: "d85b2bf8-32bc-4338-9e3e-18dfa657f136"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:39.295839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.295809 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85b2bf8-32bc-4338-9e3e-18dfa657f136-kube-api-access-ddncp" pod "d85b2bf8-32bc-4338-9e3e-18dfa657f136" (UID: "d85b2bf8-32bc-4338-9e3e-18dfa657f136"). InnerVolumeSpecName "kube-api-access-ddncp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:39.295839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.295829 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-xks-ca" pod "d85b2bf8-32bc-4338-9e3e-18dfa657f136" (UID: "d85b2bf8-32bc-4338-9e3e-18dfa657f136"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:39.394810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.394786 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:39.394810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.394812 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85b2bf8-32bc-4338-9e3e-18dfa657f136-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:39.394953 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.394822 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddncp\" (UniqueName: \"kubernetes.io/projected/d85b2bf8-32bc-4338-9e3e-18dfa657f136-kube-api-access-ddncp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:39.981521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.981492 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" Oct 31 01:27:39.981521 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.981512 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl" event={"ID":"d85b2bf8-32bc-4338-9e3e-18dfa657f136","Type":"ContainerDied","Data":"634670f23b39b4bd9d65b833553ef52b3c88186f035310db51a8cacd38bff436"} Oct 31 01:27:39.981769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:39.981589 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="634670f23b39b4bd9d65b833553ef52b3c88186f035310db51a8cacd38bff436" Oct 31 01:27:40.749067 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:40.749004 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" containerID="cri-o://82998a065ef97ecc5c91c09ec266667d279efbb9d753eec00e2fd35bfb1fb8f1" gracePeriod=45 Oct 31 01:27:40.989656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:40.989630 5719 generic.go:356] "Generic (PLEG): container finished" podID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerID="82998a065ef97ecc5c91c09ec266667d279efbb9d753eec00e2fd35bfb1fb8f1" exitCode=0 Oct 31 01:27:40.989790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:40.989724 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" event={"ID":"76372935-cf71-4ce5-865d-356fc8b8cf76","Type":"ContainerDied","Data":"82998a065ef97ecc5c91c09ec266667d279efbb9d753eec00e2fd35bfb1fb8f1"} Oct 31 01:27:40.989790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:40.989764 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" event={"ID":"76372935-cf71-4ce5-865d-356fc8b8cf76","Type":"ContainerDied","Data":"59ee3126e5176e77748083b9202a517be9ea8e50fcb125cce1053168cf7d27ce"} Oct 31 01:27:40.989790 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:40.989776 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59ee3126e5176e77748083b9202a517be9ea8e50fcb125cce1053168cf7d27ce" Oct 31 01:27:40.998967 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:40.998946 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 01:27:41.112847 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.112816 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-tmp-dir\") pod \"76372935-cf71-4ce5-865d-356fc8b8cf76\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " Oct 31 01:27:41.112973 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.112882 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76372935-cf71-4ce5-865d-356fc8b8cf76-kube-api-access-zqb65\" (UniqueName: \"kubernetes.io/projected/76372935-cf71-4ce5-865d-356fc8b8cf76-kube-api-access-zqb65\") pod \"76372935-cf71-4ce5-865d-356fc8b8cf76\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " Oct 31 01:27:41.113055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113008 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-dshm\") pod \"76372935-cf71-4ce5-865d-356fc8b8cf76\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " Oct 31 01:27:41.113118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113052 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-home\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-home\") pod \"76372935-cf71-4ce5-865d-356fc8b8cf76\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " Oct 31 01:27:41.113118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113102 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-tmp-dir" pod "76372935-cf71-4ce5-865d-356fc8b8cf76" (UID: "76372935-cf71-4ce5-865d-356fc8b8cf76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.113232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113173 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-model-cache\") pod \"76372935-cf71-4ce5-865d-356fc8b8cf76\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " Oct 31 01:27:41.113232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113206 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76372935-cf71-4ce5-865d-356fc8b8cf76-tls-certs\" (UniqueName: \"kubernetes.io/secret/76372935-cf71-4ce5-865d-356fc8b8cf76-tls-certs\") pod \"76372935-cf71-4ce5-865d-356fc8b8cf76\" (UID: \"76372935-cf71-4ce5-865d-356fc8b8cf76\") " Oct 31 01:27:41.113550 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113417 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-home" pod "76372935-cf71-4ce5-865d-356fc8b8cf76" (UID: "76372935-cf71-4ce5-865d-356fc8b8cf76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.113550 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113498 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-model-cache" pod "76372935-cf71-4ce5-865d-356fc8b8cf76" (UID: "76372935-cf71-4ce5-865d-356fc8b8cf76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.113734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113658 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.113734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113677 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.113734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.113720 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.115575 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.115547 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76372935-cf71-4ce5-865d-356fc8b8cf76-tls-certs" pod "76372935-cf71-4ce5-865d-356fc8b8cf76" (UID: "76372935-cf71-4ce5-865d-356fc8b8cf76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:41.115747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.115661 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76372935-cf71-4ce5-865d-356fc8b8cf76-kube-api-access-zqb65" pod "76372935-cf71-4ce5-865d-356fc8b8cf76" (UID: "76372935-cf71-4ce5-865d-356fc8b8cf76"). InnerVolumeSpecName "kube-api-access-zqb65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:41.115875 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.115820 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-dshm" pod "76372935-cf71-4ce5-865d-356fc8b8cf76" (UID: "76372935-cf71-4ce5-865d-356fc8b8cf76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:41.214252 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.214224 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76372935-cf71-4ce5-865d-356fc8b8cf76-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.214252 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.214252 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqb65\" (UniqueName: \"kubernetes.io/projected/76372935-cf71-4ce5-865d-356fc8b8cf76-kube-api-access-zqb65\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.214439 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.214261 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76372935-cf71-4ce5-865d-356fc8b8cf76-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:41.995715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:41.995664 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9" Oct 31 01:27:42.071890 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.071865 5719 scope.go:122] "RemoveContainer" containerID="c37db7c8b49d6814d9100493b2771e4267dea23133d2895c75a1445eb3afd372" Oct 31 01:27:42.081327 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.081311 5719 scope.go:122] "RemoveContainer" containerID="2cedbcb7c1ebaa88b7d196f7608c29c6c85680b548f19910c2691334b1b5ac90" Oct 31 01:27:42.089910 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.089865 5719 scope.go:122] "RemoveContainer" containerID="46448b549393cbcb379caae6501131109c1a4273b0948a7a697ad4a4ac9abd9c" Oct 31 01:27:42.100139 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.100108 5719 scope.go:122] "RemoveContainer" containerID="c704ce5bef67a5182892b56901b8ac1eabfb609ed15e144fd88f2ab110b47513" Oct 31 01:27:42.108556 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.108540 5719 scope.go:122] "RemoveContainer" containerID="713aac2bf73c11e9f2b8ce18ea117d898971fcf845fe452b05018f12f8efea48" Oct 31 01:27:42.116360 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.116341 5719 scope.go:122] "RemoveContainer" containerID="be2dc374b8718b86bbdf3a41e9b0a7dba4d26678d974899d95b4d49632e0d0fe" Oct 31 01:27:42.124477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.124461 5719 scope.go:122] "RemoveContainer" containerID="82998a065ef97ecc5c91c09ec266667d279efbb9d753eec00e2fd35bfb1fb8f1" Oct 31 01:27:42.158997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.158977 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9"] Oct 31 01:27:42.206085 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.206064 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-dd98f0/premium-w3-dd98f0-kserve-7f8589dcd-8wbt9"] Oct 31 01:27:42.604149 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:42.604117 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" path="/var/lib/kubelet/pods/76372935-cf71-4ce5-865d-356fc8b8cf76/volumes" Oct 31 01:27:44.532281 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:44.532253 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4"] Oct 31 01:27:44.641090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:44.641059 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f"] Oct 31 01:27:47.061660 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:47.061627 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd"] Oct 31 01:27:47.173119 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:47.173089 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d"] Oct 31 01:27:49.532811 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.532770 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" podUID="a7eb317e-2ade-4284-a448-c0bc8a06f366" containerName="payload-processing" containerID="cri-o://e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5" gracePeriod=25 Oct 31 01:27:49.641589 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.641558 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" podUID="e7512be5-ee51-4146-ae6b-615dae55118f" containerName="payload-pre-processing" containerID="cri-o://ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00" gracePeriod=25 Oct 31 01:27:49.785950 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.785899 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:27:49.891421 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.891400 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7eb317e-2ade-4284-a448-c0bc8a06f366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb317e-2ade-4284-a448-c0bc8a06f366-plugins-config-volume\") pod \"a7eb317e-2ade-4284-a448-c0bc8a06f366\" (UID: \"a7eb317e-2ade-4284-a448-c0bc8a06f366\") " Oct 31 01:27:49.891613 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.891598 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7eb317e-2ade-4284-a448-c0bc8a06f366-kube-api-access-6d6d6\" (UniqueName: \"kubernetes.io/projected/a7eb317e-2ade-4284-a448-c0bc8a06f366-kube-api-access-6d6d6\") pod \"a7eb317e-2ade-4284-a448-c0bc8a06f366\" (UID: \"a7eb317e-2ade-4284-a448-c0bc8a06f366\") " Oct 31 01:27:49.891786 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.891764 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7eb317e-2ade-4284-a448-c0bc8a06f366-plugins-config-volume" pod "a7eb317e-2ade-4284-a448-c0bc8a06f366" (UID: "a7eb317e-2ade-4284-a448-c0bc8a06f366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:49.891933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.891918 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7eb317e-2ade-4284-a448-c0bc8a06f366-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:49.893740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.893717 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7eb317e-2ade-4284-a448-c0bc8a06f366-kube-api-access-6d6d6" pod "a7eb317e-2ade-4284-a448-c0bc8a06f366" (UID: "a7eb317e-2ade-4284-a448-c0bc8a06f366"). InnerVolumeSpecName "kube-api-access-6d6d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:49.898425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.898403 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:27:49.992331 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.992304 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7512be5-ee51-4146-ae6b-615dae55118f-kube-api-access-vswbr\" (UniqueName: \"kubernetes.io/projected/e7512be5-ee51-4146-ae6b-615dae55118f-kube-api-access-vswbr\") pod \"e7512be5-ee51-4146-ae6b-615dae55118f\" (UID: \"e7512be5-ee51-4146-ae6b-615dae55118f\") " Oct 31 01:27:49.992469 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.992347 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7512be5-ee51-4146-ae6b-615dae55118f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7512be5-ee51-4146-ae6b-615dae55118f-plugins-config-volume\") pod \"e7512be5-ee51-4146-ae6b-615dae55118f\" (UID: \"e7512be5-ee51-4146-ae6b-615dae55118f\") " Oct 31 01:27:49.992740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.992719 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7512be5-ee51-4146-ae6b-615dae55118f-plugins-config-volume" pod "e7512be5-ee51-4146-ae6b-615dae55118f" (UID: "e7512be5-ee51-4146-ae6b-615dae55118f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:49.992880 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.992868 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7512be5-ee51-4146-ae6b-615dae55118f-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:49.992933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.992883 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6d6\" (UniqueName: \"kubernetes.io/projected/a7eb317e-2ade-4284-a448-c0bc8a06f366-kube-api-access-6d6d6\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:49.994318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:49.994301 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7512be5-ee51-4146-ae6b-615dae55118f-kube-api-access-vswbr" pod "e7512be5-ee51-4146-ae6b-615dae55118f" (UID: "e7512be5-ee51-4146-ae6b-615dae55118f"). InnerVolumeSpecName "kube-api-access-vswbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:50.045347 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.045283 5719 generic.go:356] "Generic (PLEG): container finished" podID="a7eb317e-2ade-4284-a448-c0bc8a06f366" containerID="e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5" exitCode=0 Oct 31 01:27:50.045347 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.045329 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" Oct 31 01:27:50.045538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.045366 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" event={"ID":"a7eb317e-2ade-4284-a448-c0bc8a06f366","Type":"ContainerDied","Data":"e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5"} Oct 31 01:27:50.045538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.045444 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4" event={"ID":"a7eb317e-2ade-4284-a448-c0bc8a06f366","Type":"ContainerDied","Data":"d75e14218857e12de1e40ce53d4e4b9edc87ff46b647cc186d6a7d3a970ddfcc"} Oct 31 01:27:50.045538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.045479 5719 scope.go:122] "RemoveContainer" containerID="e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5" Oct 31 01:27:50.046896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.046874 5719 generic.go:356] "Generic (PLEG): container finished" podID="e7512be5-ee51-4146-ae6b-615dae55118f" containerID="ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00" exitCode=0 Oct 31 01:27:50.047009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.046912 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" Oct 31 01:27:50.047009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.046925 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" event={"ID":"e7512be5-ee51-4146-ae6b-615dae55118f","Type":"ContainerDied","Data":"ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00"} Oct 31 01:27:50.047009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.046966 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f" event={"ID":"e7512be5-ee51-4146-ae6b-615dae55118f","Type":"ContainerDied","Data":"4ca4bf49fc9715c6367e73aae029627d8bd9b23e0ab9d5e3a88813a2e3e95ef8"} Oct 31 01:27:50.056430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.056381 5719 scope.go:122] "RemoveContainer" containerID="e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5" Oct 31 01:27:50.056657 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:27:50.056635 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5\": container with ID starting with e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5 not found: ID does not exist" containerID="e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5" Oct 31 01:27:50.056755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.056669 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5"} err="failed to get container status \"e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5\": rpc error: code = NotFound desc = could not find container \"e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5\": container with ID starting with e405de8e4c28a3c6f997ba23d4ea4288666c1646ff24eaae21b49cad45fcb3b5 not found: ID does not exist" Oct 31 01:27:50.056755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.056720 5719 scope.go:122] "RemoveContainer" containerID="ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00" Oct 31 01:27:50.065325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.065305 5719 scope.go:122] "RemoveContainer" containerID="ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00" Oct 31 01:27:50.065568 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:27:50.065551 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00\": container with ID starting with ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00 not found: ID does not exist" containerID="ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00" Oct 31 01:27:50.065625 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.065574 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00"} err="failed to get container status \"ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00\": rpc error: code = NotFound desc = could not find container \"ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00\": container with ID starting with ea8e25924d4bd8bb84aeb83e5a3641b33931a178e43237b7b7e3b5ae04f3bb00 not found: ID does not exist" Oct 31 01:27:50.093477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.093458 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vswbr\" (UniqueName: \"kubernetes.io/projected/e7512be5-ee51-4146-ae6b-615dae55118f-kube-api-access-vswbr\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:27:50.203338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.203315 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f"] Oct 31 01:27:50.207343 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.207316 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9332a5-5b55b4bff9-l4x5f"] Oct 31 01:27:50.311880 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.311813 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4"] Oct 31 01:27:50.316259 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.316237 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9332a5-b66fd5659-2ddd4"] Oct 31 01:27:50.603007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.602977 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7eb317e-2ade-4284-a448-c0bc8a06f366" path="/var/lib/kubelet/pods/a7eb317e-2ade-4284-a448-c0bc8a06f366/volumes" Oct 31 01:27:50.603533 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:50.603514 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7512be5-ee51-4146-ae6b-615dae55118f" path="/var/lib/kubelet/pods/e7512be5-ee51-4146-ae6b-615dae55118f/volumes" Oct 31 01:27:52.062859 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:52.062825 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" podUID="fe5ce7c3-62bb-4abe-b797-465f984d022a" containerName="payload-processing" containerID="cri-o://0d53012c636626005da9e2e25f6b14a96b587227fc67343cf15c574e3c6327f6" gracePeriod=25 Oct 31 01:27:52.173474 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:52.173442 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" podUID="4704dfad-d066-4b5d-a97c-daed41f983f7" containerName="payload-pre-processing" containerID="cri-o://ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493" gracePeriod=25 Oct 31 01:27:56.719612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:56.719575 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" podUID="fe5ce7c3-62bb-4abe-b797-465f984d022a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:27:56.720005 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:27:56.719574 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" podUID="4704dfad-d066-4b5d-a97c-daed41f983f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:28:01.889347 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:01.889311 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p"] Oct 31 01:28:01.936454 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:01.936425 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ba34eb-ws64p"] Oct 31 01:28:02.603668 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:02.603633 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0a0594-3186-487f-a948-92519910593c" path="/var/lib/kubelet/pods/9e0a0594-3186-487f-a948-92519910593c/volumes" Oct 31 01:28:03.011820 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:28:03.011735 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:28:06.719943 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:06.719912 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" podUID="fe5ce7c3-62bb-4abe-b797-465f984d022a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:28:06.719943 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:06.719936 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" podUID="4704dfad-d066-4b5d-a97c-daed41f983f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:28:10.065655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.065624 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct"] Oct 31 01:28:10.066180 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066165 5719 cpu_manager.go:414] "containerMap: removing container" podUID="011e3744-fcb9-4729-8882-c8c42797b6dd" containerName="revoke-keys" Oct 31 01:28:10.066180 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066181 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011e3744-fcb9-4729-8882-c8c42797b6dd" containerName="revoke-keys" Oct 31 01:28:10.066273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066197 5719 cpu_manager.go:414] "containerMap: removing container" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" Oct 31 01:28:10.066273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066202 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" Oct 31 01:28:10.066273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066211 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e7512be5-ee51-4146-ae6b-615dae55118f" containerName="payload-pre-processing" Oct 31 01:28:10.066273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066216 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7512be5-ee51-4146-ae6b-615dae55118f" containerName="payload-pre-processing" Oct 31 01:28:10.066273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066227 5719 cpu_manager.go:414] "containerMap: removing container" podUID="a7eb317e-2ade-4284-a448-c0bc8a06f366" containerName="payload-processing" Oct 31 01:28:10.066273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066231 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7eb317e-2ade-4284-a448-c0bc8a06f366" containerName="payload-processing" Oct 31 01:28:10.066273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066240 5719 cpu_manager.go:414] "containerMap: removing container" podUID="d85b2bf8-32bc-4338-9e3e-18dfa657f136" containerName="revoke-keys" Oct 31 01:28:10.066273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066244 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85b2bf8-32bc-4338-9e3e-18dfa657f136" containerName="revoke-keys" Oct 31 01:28:10.066522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066318 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="011e3744-fcb9-4729-8882-c8c42797b6dd" containerName="revoke-keys" Oct 31 01:28:10.066522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066326 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="76372935-cf71-4ce5-865d-356fc8b8cf76" containerName="main" Oct 31 01:28:10.066522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066331 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7512be5-ee51-4146-ae6b-615dae55118f" containerName="payload-pre-processing" Oct 31 01:28:10.066522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066338 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7eb317e-2ade-4284-a448-c0bc8a06f366" containerName="payload-processing" Oct 31 01:28:10.066522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.066345 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85b2bf8-32bc-4338-9e3e-18dfa657f136" containerName="revoke-keys" Oct 31 01:28:10.069643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.069625 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.121401 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.121380 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:10.124971 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.124947 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct"] Oct 31 01:28:10.128090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.128057 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:28:10.169319 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.169298 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:28:10.169319 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.169313 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:28:10.169559 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.169459 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:28:10.190573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.190546 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ssbc\" (UniqueName: \"kubernetes.io/projected/856d1257-dc8a-47d1-a61c-66b8f1615366-kube-api-access-8ssbc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.190717 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.190666 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.190775 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.190720 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.291538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.291512 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssbc\" (UniqueName: \"kubernetes.io/projected/856d1257-dc8a-47d1-a61c-66b8f1615366-kube-api-access-8ssbc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.291653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.291576 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.291653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.291597 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.292409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.292335 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.294139 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.294119 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.350028 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.349980 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ssbc\" (UniqueName: \"kubernetes.io/projected/856d1257-dc8a-47d1-a61c-66b8f1615366-kube-api-access-8ssbc\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.382806 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.382781 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:10.758424 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:10.758324 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct"] Oct 31 01:28:10.767386 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:28:10.767352 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856d1257_dc8a_47d1_a61c_66b8f1615366.slice/crio-514bc7ffef8b0db00cba57d6af215ec161740235264bbf6fb5e8b4f86da199b4 WatchSource:0}: Error finding container 514bc7ffef8b0db00cba57d6af215ec161740235264bbf6fb5e8b4f86da199b4: Status 404 returned error can't find the container with id 514bc7ffef8b0db00cba57d6af215ec161740235264bbf6fb5e8b4f86da199b4 Oct 31 01:28:11.178770 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:11.178734 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" event={"ID":"856d1257-dc8a-47d1-a61c-66b8f1615366","Type":"ContainerStarted","Data":"aadfb99169c3c6c616483fd2b89b89ecd74a5c688a58a6fb0b0854380e90025b"} Oct 31 01:28:11.179207 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:11.178790 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" event={"ID":"856d1257-dc8a-47d1-a61c-66b8f1615366","Type":"ContainerStarted","Data":"514bc7ffef8b0db00cba57d6af215ec161740235264bbf6fb5e8b4f86da199b4"} Oct 31 01:28:11.292354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:11.292276 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" podStartSLOduration=1.292262381 podStartE2EDuration="1.292262381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:11.290375093 +0000 UTC m=+3572.826517400" watchObservedRunningTime="2026-10-31 01:28:11.292262381 +0000 UTC m=+3572.828404691" Oct 31 01:28:12.187098 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.187072 5719 generic.go:356] "Generic (PLEG): container finished" podID="fe5ce7c3-62bb-4abe-b797-465f984d022a" containerID="0d53012c636626005da9e2e25f6b14a96b587227fc67343cf15c574e3c6327f6" exitCode=0 Oct 31 01:28:12.187472 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.187229 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" event={"ID":"fe5ce7c3-62bb-4abe-b797-465f984d022a","Type":"ContainerDied","Data":"0d53012c636626005da9e2e25f6b14a96b587227fc67343cf15c574e3c6327f6"} Oct 31 01:28:12.199784 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.199754 5719 generic.go:356] "Generic (PLEG): container finished" podID="856d1257-dc8a-47d1-a61c-66b8f1615366" containerID="aadfb99169c3c6c616483fd2b89b89ecd74a5c688a58a6fb0b0854380e90025b" exitCode=0 Oct 31 01:28:12.199914 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.199798 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" event={"ID":"856d1257-dc8a-47d1-a61c-66b8f1615366","Type":"ContainerDied","Data":"aadfb99169c3c6c616483fd2b89b89ecd74a5c688a58a6fb0b0854380e90025b"} Oct 31 01:28:12.333031 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.332968 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:28:12.413878 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.413850 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5ce7c3-62bb-4abe-b797-465f984d022a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ce7c3-62bb-4abe-b797-465f984d022a-plugins-config-volume\") pod \"fe5ce7c3-62bb-4abe-b797-465f984d022a\" (UID: \"fe5ce7c3-62bb-4abe-b797-465f984d022a\") " Oct 31 01:28:12.414047 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.414015 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5ce7c3-62bb-4abe-b797-465f984d022a-kube-api-access-9ng9f\" (UniqueName: \"kubernetes.io/projected/fe5ce7c3-62bb-4abe-b797-465f984d022a-kube-api-access-9ng9f\") pod \"fe5ce7c3-62bb-4abe-b797-465f984d022a\" (UID: \"fe5ce7c3-62bb-4abe-b797-465f984d022a\") " Oct 31 01:28:12.414238 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.414216 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ce7c3-62bb-4abe-b797-465f984d022a-plugins-config-volume" pod "fe5ce7c3-62bb-4abe-b797-465f984d022a" (UID: "fe5ce7c3-62bb-4abe-b797-465f984d022a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:12.414458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.414434 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe5ce7c3-62bb-4abe-b797-465f984d022a-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:12.416127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.416105 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ce7c3-62bb-4abe-b797-465f984d022a-kube-api-access-9ng9f" pod "fe5ce7c3-62bb-4abe-b797-465f984d022a" (UID: "fe5ce7c3-62bb-4abe-b797-465f984d022a"). InnerVolumeSpecName "kube-api-access-9ng9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:12.435743 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.435724 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:28:12.515673 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.515644 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4704dfad-d066-4b5d-a97c-daed41f983f7-kube-api-access-wrvdl\" (UniqueName: \"kubernetes.io/projected/4704dfad-d066-4b5d-a97c-daed41f983f7-kube-api-access-wrvdl\") pod \"4704dfad-d066-4b5d-a97c-daed41f983f7\" (UID: \"4704dfad-d066-4b5d-a97c-daed41f983f7\") " Oct 31 01:28:12.515844 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.515727 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4704dfad-d066-4b5d-a97c-daed41f983f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4704dfad-d066-4b5d-a97c-daed41f983f7-plugins-config-volume\") pod \"4704dfad-d066-4b5d-a97c-daed41f983f7\" (UID: \"4704dfad-d066-4b5d-a97c-daed41f983f7\") " Oct 31 01:28:12.516028 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.516014 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ng9f\" (UniqueName: \"kubernetes.io/projected/fe5ce7c3-62bb-4abe-b797-465f984d022a-kube-api-access-9ng9f\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:12.516191 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.516165 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4704dfad-d066-4b5d-a97c-daed41f983f7-plugins-config-volume" pod "4704dfad-d066-4b5d-a97c-daed41f983f7" (UID: "4704dfad-d066-4b5d-a97c-daed41f983f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:12.518064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.518036 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4704dfad-d066-4b5d-a97c-daed41f983f7-kube-api-access-wrvdl" pod "4704dfad-d066-4b5d-a97c-daed41f983f7" (UID: "4704dfad-d066-4b5d-a97c-daed41f983f7"). InnerVolumeSpecName "kube-api-access-wrvdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:12.616597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.616573 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrvdl\" (UniqueName: \"kubernetes.io/projected/4704dfad-d066-4b5d-a97c-daed41f983f7-kube-api-access-wrvdl\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:12.616597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:12.616598 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4704dfad-d066-4b5d-a97c-daed41f983f7-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:13.206580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.206547 5719 generic.go:356] "Generic (PLEG): container finished" podID="4704dfad-d066-4b5d-a97c-daed41f983f7" containerID="ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493" exitCode=0 Oct 31 01:28:13.207029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.206596 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" Oct 31 01:28:13.207029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.206625 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" event={"ID":"4704dfad-d066-4b5d-a97c-daed41f983f7","Type":"ContainerDied","Data":"ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493"} Oct 31 01:28:13.207029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.206680 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d" event={"ID":"4704dfad-d066-4b5d-a97c-daed41f983f7","Type":"ContainerDied","Data":"afa02207b9b0813f70ad49eef0a3f8ed5af7fded603da7457dece629e25c3f02"} Oct 31 01:28:13.207029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.206725 5719 scope.go:122] "RemoveContainer" containerID="ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493" Oct 31 01:28:13.208361 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.208334 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" Oct 31 01:28:13.208481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.208383 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd" event={"ID":"fe5ce7c3-62bb-4abe-b797-465f984d022a","Type":"ContainerDied","Data":"1fdf8d36a8c5c8c8c39a2a38fb5398d1c8e02ac6b66b81efc7f662e2feca15f1"} Oct 31 01:28:13.217062 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.217028 5719 scope.go:122] "RemoveContainer" containerID="ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493" Oct 31 01:28:13.217347 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:28:13.217329 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493\": container with ID starting with ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493 not found: ID does not exist" containerID="ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493" Oct 31 01:28:13.217428 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.217355 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493"} err="failed to get container status \"ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493\": rpc error: code = NotFound desc = could not find container \"ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493\": container with ID starting with ccfb134b31712c871c5d91a9971ac1b8ca0ed219e0e6741324c25a882fabc493 not found: ID does not exist" Oct 31 01:28:13.217428 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.217376 5719 scope.go:122] "RemoveContainer" containerID="0d53012c636626005da9e2e25f6b14a96b587227fc67343cf15c574e3c6327f6" Oct 31 01:28:13.354275 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.354253 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:13.365338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.365316 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d"] Oct 31 01:28:13.369455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.369432 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4397fc-6bd899b57f-ghm7d"] Oct 31 01:28:13.473968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.473913 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd"] Oct 31 01:28:13.478022 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.477999 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4397fc-86dfb8f9c-g8gwd"] Oct 31 01:28:13.523768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.523749 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-openshift-ca\") pod \"856d1257-dc8a-47d1-a61c-66b8f1615366\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " Oct 31 01:28:13.523893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.523788 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-xks-ca\") pod \"856d1257-dc8a-47d1-a61c-66b8f1615366\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " Oct 31 01:28:13.523893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.523822 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856d1257-dc8a-47d1-a61c-66b8f1615366-kube-api-access-8ssbc\" (UniqueName: \"kubernetes.io/projected/856d1257-dc8a-47d1-a61c-66b8f1615366-kube-api-access-8ssbc\") pod \"856d1257-dc8a-47d1-a61c-66b8f1615366\" (UID: \"856d1257-dc8a-47d1-a61c-66b8f1615366\") " Oct 31 01:28:13.524126 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.524100 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-openshift-ca" pod "856d1257-dc8a-47d1-a61c-66b8f1615366" (UID: "856d1257-dc8a-47d1-a61c-66b8f1615366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:13.524235 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.524222 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:13.526081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.526053 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-xks-ca" pod "856d1257-dc8a-47d1-a61c-66b8f1615366" (UID: "856d1257-dc8a-47d1-a61c-66b8f1615366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:13.526195 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.526181 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856d1257-dc8a-47d1-a61c-66b8f1615366-kube-api-access-8ssbc" pod "856d1257-dc8a-47d1-a61c-66b8f1615366" (UID: "856d1257-dc8a-47d1-a61c-66b8f1615366"). InnerVolumeSpecName "kube-api-access-8ssbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:13.625367 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.625338 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/856d1257-dc8a-47d1-a61c-66b8f1615366-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:13.625367 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:13.625364 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ssbc\" (UniqueName: \"kubernetes.io/projected/856d1257-dc8a-47d1-a61c-66b8f1615366-kube-api-access-8ssbc\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:14.216463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:14.216436 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" Oct 31 01:28:14.216915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:14.216426 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct" event={"ID":"856d1257-dc8a-47d1-a61c-66b8f1615366","Type":"ContainerDied","Data":"514bc7ffef8b0db00cba57d6af215ec161740235264bbf6fb5e8b4f86da199b4"} Oct 31 01:28:14.216915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:14.216578 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="514bc7ffef8b0db00cba57d6af215ec161740235264bbf6fb5e8b4f86da199b4" Oct 31 01:28:14.602585 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:14.602556 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4704dfad-d066-4b5d-a97c-daed41f983f7" path="/var/lib/kubelet/pods/4704dfad-d066-4b5d-a97c-daed41f983f7/volumes" Oct 31 01:28:14.602997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:14.602982 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5ce7c3-62bb-4abe-b797-465f984d022a" path="/var/lib/kubelet/pods/fe5ce7c3-62bb-4abe-b797-465f984d022a/volumes" Oct 31 01:28:21.543160 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:21.543128 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf"] Oct 31 01:28:21.651526 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:21.651497 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s"] Oct 31 01:28:26.544133 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:26.544048 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" podUID="9e30dbf8-db51-4772-9f60-852d38b8d182" containerName="payload-processing" containerID="cri-o://647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510" gracePeriod=25 Oct 31 01:28:26.652748 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:26.652680 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" podUID="61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" containerName="payload-pre-processing" containerID="cri-o://95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386" gracePeriod=25 Oct 31 01:28:27.812726 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.812674 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx"] Oct 31 01:28:27.813514 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813494 5719 cpu_manager.go:414] "containerMap: removing container" podUID="4704dfad-d066-4b5d-a97c-daed41f983f7" containerName="payload-pre-processing" Oct 31 01:28:27.813626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813516 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4704dfad-d066-4b5d-a97c-daed41f983f7" containerName="payload-pre-processing" Oct 31 01:28:27.813626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813557 5719 cpu_manager.go:414] "containerMap: removing container" podUID="856d1257-dc8a-47d1-a61c-66b8f1615366" containerName="revoke-keys" Oct 31 01:28:27.813626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813562 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856d1257-dc8a-47d1-a61c-66b8f1615366" containerName="revoke-keys" Oct 31 01:28:27.813626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813570 5719 cpu_manager.go:414] "containerMap: removing container" podUID="fe5ce7c3-62bb-4abe-b797-465f984d022a" containerName="payload-processing" Oct 31 01:28:27.813626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813574 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5ce7c3-62bb-4abe-b797-465f984d022a" containerName="payload-processing" Oct 31 01:28:27.813829 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813654 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="4704dfad-d066-4b5d-a97c-daed41f983f7" containerName="payload-pre-processing" Oct 31 01:28:27.813829 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813669 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="856d1257-dc8a-47d1-a61c-66b8f1615366" containerName="revoke-keys" Oct 31 01:28:27.813829 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.813674 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5ce7c3-62bb-4abe-b797-465f984d022a" containerName="payload-processing" Oct 31 01:28:27.818352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.818337 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:27.858785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.858761 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65klw\" (UniqueName: \"kubernetes.io/projected/35093d63-9a7e-4570-90c9-cdc67f08bde4-kube-api-access-65klw\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:27.858956 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.858868 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:27.859017 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.858967 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:27.872039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.872012 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j"] Oct 31 01:28:27.876563 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.876540 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx"] Oct 31 01:28:27.876661 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.876630 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:27.916838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.916814 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:27.916982 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.916965 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:28:27.917035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.916981 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:28:27.917035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.917013 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:28:27.932887 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.932863 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j"] Oct 31 01:28:27.959775 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.959753 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:27.959917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.959821 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:27.959917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.959892 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98s65\" (UniqueName: \"kubernetes.io/projected/4b66fec2-399e-4fd6-9118-a74cb4f92c68-kube-api-access-98s65\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:27.960039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.960004 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:27.960102 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.960054 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:27.960165 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.960138 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65klw\" (UniqueName: \"kubernetes.io/projected/35093d63-9a7e-4570-90c9-cdc67f08bde4-kube-api-access-65klw\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:27.960735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.960716 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:27.962425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:27.962401 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:28.019406 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.019383 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65klw\" (UniqueName: \"kubernetes.io/projected/35093d63-9a7e-4570-90c9-cdc67f08bde4-kube-api-access-65klw\") pod \"maas-api-revoke-sub-cc24400e-qr4wx\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:28.060964 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.060935 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:28.061083 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.061021 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:28.061083 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.061078 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98s65\" (UniqueName: \"kubernetes.io/projected/4b66fec2-399e-4fd6-9118-a74cb4f92c68-kube-api-access-98s65\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:28.061553 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.061525 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:28.063432 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.063374 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:28.120060 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.120034 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98s65\" (UniqueName: \"kubernetes.io/projected/4b66fec2-399e-4fd6-9118-a74cb4f92c68-kube-api-access-98s65\") pod \"maas-api-revoke-sub-e5be7dc2-pmp5j\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:28.131729 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.131709 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:28.190196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.189940 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:28.507440 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.507410 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx"] Oct 31 01:28:28.513358 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:28:28.513331 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35093d63_9a7e_4570_90c9_cdc67f08bde4.slice/crio-61ca17d3f8ccecadde30effbc22825937e1ecd2b6a4a3a261d14fde8ed07e599 WatchSource:0}: Error finding container 61ca17d3f8ccecadde30effbc22825937e1ecd2b6a4a3a261d14fde8ed07e599: Status 404 returned error can't find the container with id 61ca17d3f8ccecadde30effbc22825937e1ecd2b6a4a3a261d14fde8ed07e599 Oct 31 01:28:28.570718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.570664 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j"] Oct 31 01:28:28.576641 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:28:28.576618 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b66fec2_399e_4fd6_9118_a74cb4f92c68.slice/crio-dbad7a28c094e89a815dd08cbac53d7b5b9b9dc80e6e3947e0e023b92140fb2e WatchSource:0}: Error finding container dbad7a28c094e89a815dd08cbac53d7b5b9b9dc80e6e3947e0e023b92140fb2e: Status 404 returned error can't find the container with id dbad7a28c094e89a815dd08cbac53d7b5b9b9dc80e6e3947e0e023b92140fb2e Oct 31 01:28:28.693747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.693721 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl"] Oct 31 01:28:28.696570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:28.696547 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4397fc-bj4nl"] Oct 31 01:28:29.308108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:29.308065 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" event={"ID":"4b66fec2-399e-4fd6-9118-a74cb4f92c68","Type":"ContainerStarted","Data":"4416a09eff9863dcfea958f023edac33645b46c2ad6bcacadca79cc348b6c15d"} Oct 31 01:28:29.308529 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:29.308163 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" event={"ID":"4b66fec2-399e-4fd6-9118-a74cb4f92c68","Type":"ContainerStarted","Data":"dbad7a28c094e89a815dd08cbac53d7b5b9b9dc80e6e3947e0e023b92140fb2e"} Oct 31 01:28:29.309777 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:29.309750 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" event={"ID":"35093d63-9a7e-4570-90c9-cdc67f08bde4","Type":"ContainerStarted","Data":"7e9e0706db08fc666eb7d0ba7f7eb23a8e7ff93cdb192f1880c9e38f511512de"} Oct 31 01:28:29.309900 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:29.309799 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" event={"ID":"35093d63-9a7e-4570-90c9-cdc67f08bde4","Type":"ContainerStarted","Data":"61ca17d3f8ccecadde30effbc22825937e1ecd2b6a4a3a261d14fde8ed07e599"} Oct 31 01:28:29.421203 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:29.421157 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" podStartSLOduration=2.421144448 podStartE2EDuration="2.421144448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:29.419285488 +0000 UTC m=+3590.955427794" watchObservedRunningTime="2026-10-31 01:28:29.421144448 +0000 UTC m=+3590.957286787" Oct 31 01:28:29.534553 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:29.534500 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" podStartSLOduration=2.534485254 podStartE2EDuration="2.534485254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:29.532336469 +0000 UTC m=+3591.068478793" watchObservedRunningTime="2026-10-31 01:28:29.534485254 +0000 UTC m=+3591.070627559" Oct 31 01:28:30.319679 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:30.319642 5719 generic.go:356] "Generic (PLEG): container finished" podID="4b66fec2-399e-4fd6-9118-a74cb4f92c68" containerID="4416a09eff9863dcfea958f023edac33645b46c2ad6bcacadca79cc348b6c15d" exitCode=0 Oct 31 01:28:30.320140 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:30.319727 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" event={"ID":"4b66fec2-399e-4fd6-9118-a74cb4f92c68","Type":"ContainerDied","Data":"4416a09eff9863dcfea958f023edac33645b46c2ad6bcacadca79cc348b6c15d"} Oct 31 01:28:30.321488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:30.321463 5719 generic.go:356] "Generic (PLEG): container finished" podID="35093d63-9a7e-4570-90c9-cdc67f08bde4" containerID="7e9e0706db08fc666eb7d0ba7f7eb23a8e7ff93cdb192f1880c9e38f511512de" exitCode=0 Oct 31 01:28:30.321612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:30.321509 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" event={"ID":"35093d63-9a7e-4570-90c9-cdc67f08bde4","Type":"ContainerDied","Data":"7e9e0706db08fc666eb7d0ba7f7eb23a8e7ff93cdb192f1880c9e38f511512de"} Oct 31 01:28:30.602872 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:30.602804 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85b2bf8-32bc-4338-9e3e-18dfa657f136" path="/var/lib/kubelet/pods/d85b2bf8-32bc-4338-9e3e-18dfa657f136/volumes" Oct 31 01:28:31.481388 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.481359 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:31.503922 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.503885 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" podUID="9e30dbf8-db51-4772-9f60-852d38b8d182" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:28:31.504263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.504238 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" podUID="61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:28:31.506408 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.506391 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:31.602452 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.602390 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-openshift-ca\") pod \"35093d63-9a7e-4570-90c9-cdc67f08bde4\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " Oct 31 01:28:31.602452 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.602430 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b66fec2-399e-4fd6-9118-a74cb4f92c68-kube-api-access-98s65\" (UniqueName: \"kubernetes.io/projected/4b66fec2-399e-4fd6-9118-a74cb4f92c68-kube-api-access-98s65\") pod \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " Oct 31 01:28:31.602644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.602476 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-openshift-ca\") pod \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " Oct 31 01:28:31.602644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.602579 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35093d63-9a7e-4570-90c9-cdc67f08bde4-kube-api-access-65klw\" (UniqueName: \"kubernetes.io/projected/35093d63-9a7e-4570-90c9-cdc67f08bde4-kube-api-access-65klw\") pod \"35093d63-9a7e-4570-90c9-cdc67f08bde4\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " Oct 31 01:28:31.602780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.602663 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-xks-ca\") pod \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\" (UID: \"4b66fec2-399e-4fd6-9118-a74cb4f92c68\") " Oct 31 01:28:31.602780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.602763 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-xks-ca\") pod \"35093d63-9a7e-4570-90c9-cdc67f08bde4\" (UID: \"35093d63-9a7e-4570-90c9-cdc67f08bde4\") " Oct 31 01:28:31.602909 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.602785 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-openshift-ca" pod "35093d63-9a7e-4570-90c9-cdc67f08bde4" (UID: "35093d63-9a7e-4570-90c9-cdc67f08bde4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:31.603188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.603163 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:31.603404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.603367 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-openshift-ca" pod "4b66fec2-399e-4fd6-9118-a74cb4f92c68" (UID: "4b66fec2-399e-4fd6-9118-a74cb4f92c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:31.605133 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.605107 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35093d63-9a7e-4570-90c9-cdc67f08bde4-kube-api-access-65klw" pod "35093d63-9a7e-4570-90c9-cdc67f08bde4" (UID: "35093d63-9a7e-4570-90c9-cdc67f08bde4"). InnerVolumeSpecName "kube-api-access-65klw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:31.605755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.605729 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-xks-ca" pod "4b66fec2-399e-4fd6-9118-a74cb4f92c68" (UID: "4b66fec2-399e-4fd6-9118-a74cb4f92c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:31.605884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.605864 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-xks-ca" pod "35093d63-9a7e-4570-90c9-cdc67f08bde4" (UID: "35093d63-9a7e-4570-90c9-cdc67f08bde4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:31.606047 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.606026 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b66fec2-399e-4fd6-9118-a74cb4f92c68-kube-api-access-98s65" pod "4b66fec2-399e-4fd6-9118-a74cb4f92c68" (UID: "4b66fec2-399e-4fd6-9118-a74cb4f92c68"). InnerVolumeSpecName "kube-api-access-98s65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:31.704340 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.704313 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:31.704340 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.704335 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65klw\" (UniqueName: \"kubernetes.io/projected/35093d63-9a7e-4570-90c9-cdc67f08bde4-kube-api-access-65klw\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:31.704509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.704346 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b66fec2-399e-4fd6-9118-a74cb4f92c68-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:31.704509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.704354 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35093d63-9a7e-4570-90c9-cdc67f08bde4-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:31.704509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:31.704364 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98s65\" (UniqueName: \"kubernetes.io/projected/4b66fec2-399e-4fd6-9118-a74cb4f92c68-kube-api-access-98s65\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:32.334895 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:32.334868 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" Oct 31 01:28:32.335074 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:32.334872 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx" event={"ID":"35093d63-9a7e-4570-90c9-cdc67f08bde4","Type":"ContainerDied","Data":"61ca17d3f8ccecadde30effbc22825937e1ecd2b6a4a3a261d14fde8ed07e599"} Oct 31 01:28:32.335074 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:32.334989 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ca17d3f8ccecadde30effbc22825937e1ecd2b6a4a3a261d14fde8ed07e599" Oct 31 01:28:32.336547 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:32.336519 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" event={"ID":"4b66fec2-399e-4fd6-9118-a74cb4f92c68","Type":"ContainerDied","Data":"dbad7a28c094e89a815dd08cbac53d7b5b9b9dc80e6e3947e0e023b92140fb2e"} Oct 31 01:28:32.336679 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:32.336556 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j" Oct 31 01:28:32.336679 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:32.336560 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbad7a28c094e89a815dd08cbac53d7b5b9b9dc80e6e3947e0e023b92140fb2e" Oct 31 01:28:33.008977 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:28:33.008942 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:28:36.785857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.785822 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k"] Oct 31 01:28:36.786354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.786340 5719 cpu_manager.go:414] "containerMap: removing container" podUID="4b66fec2-399e-4fd6-9118-a74cb4f92c68" containerName="revoke-keys" Oct 31 01:28:36.786410 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.786356 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b66fec2-399e-4fd6-9118-a74cb4f92c68" containerName="revoke-keys" Oct 31 01:28:36.786410 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.786399 5719 cpu_manager.go:414] "containerMap: removing container" podUID="35093d63-9a7e-4570-90c9-cdc67f08bde4" containerName="revoke-keys" Oct 31 01:28:36.786410 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.786404 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35093d63-9a7e-4570-90c9-cdc67f08bde4" containerName="revoke-keys" Oct 31 01:28:36.786517 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.786487 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="35093d63-9a7e-4570-90c9-cdc67f08bde4" containerName="revoke-keys" Oct 31 01:28:36.786517 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.786502 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b66fec2-399e-4fd6-9118-a74cb4f92c68" containerName="revoke-keys" Oct 31 01:28:36.790078 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.790062 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:36.841586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.841561 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:36.841586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.841572 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:28:36.852787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.852762 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:36.852901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.852801 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:36.852901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.852891 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/578a76ef-aa57-4e07-ae0c-935ef3cc2243-kube-api-access-2d7bz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:36.889414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.889391 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:28:36.889414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.889391 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:28:36.892001 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.891984 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k"] Oct 31 01:28:36.953568 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.953540 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:36.953681 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.953571 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:36.953681 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.953594 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/578a76ef-aa57-4e07-ae0c-935ef3cc2243-kube-api-access-2d7bz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:36.954325 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.954305 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:36.956118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:36.956098 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:37.012644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:37.012614 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/578a76ef-aa57-4e07-ae0c-935ef3cc2243-kube-api-access-2d7bz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:37.103073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:37.103054 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:37.479301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:37.479231 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k"] Oct 31 01:28:37.485280 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:28:37.485254 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578a76ef_aa57_4e07_ae0c_935ef3cc2243.slice/crio-6cc3e7b012eda6e3a2dc8b10e2e690998efe8d75558259b01cb33e893dbec8ef WatchSource:0}: Error finding container 6cc3e7b012eda6e3a2dc8b10e2e690998efe8d75558259b01cb33e893dbec8ef: Status 404 returned error can't find the container with id 6cc3e7b012eda6e3a2dc8b10e2e690998efe8d75558259b01cb33e893dbec8ef Oct 31 01:28:38.373514 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:38.373477 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" event={"ID":"578a76ef-aa57-4e07-ae0c-935ef3cc2243","Type":"ContainerStarted","Data":"9f1851ad9306a12a4ea5926f55aeede88783047e051a0e146d2e8d7b33e48fed"} Oct 31 01:28:38.374014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:38.373568 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" event={"ID":"578a76ef-aa57-4e07-ae0c-935ef3cc2243","Type":"ContainerStarted","Data":"6cc3e7b012eda6e3a2dc8b10e2e690998efe8d75558259b01cb33e893dbec8ef"} Oct 31 01:28:38.485978 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:38.485940 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" podStartSLOduration=2.485927128 podStartE2EDuration="2.485927128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:38.485057471 +0000 UTC m=+3600.021199775" watchObservedRunningTime="2026-10-31 01:28:38.485927128 +0000 UTC m=+3600.022069432" Oct 31 01:28:39.367825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:39.367787 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:28:39.380552 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:39.380522 5719 generic.go:356] "Generic (PLEG): container finished" podID="578a76ef-aa57-4e07-ae0c-935ef3cc2243" containerID="9f1851ad9306a12a4ea5926f55aeede88783047e051a0e146d2e8d7b33e48fed" exitCode=0 Oct 31 01:28:39.380894 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:39.380598 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" event={"ID":"578a76ef-aa57-4e07-ae0c-935ef3cc2243","Type":"ContainerDied","Data":"9f1851ad9306a12a4ea5926f55aeede88783047e051a0e146d2e8d7b33e48fed"} Oct 31 01:28:40.540609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.540581 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:40.690352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.690284 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-xks-ca\") pod \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " Oct 31 01:28:40.690484 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.690425 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578a76ef-aa57-4e07-ae0c-935ef3cc2243-kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/578a76ef-aa57-4e07-ae0c-935ef3cc2243-kube-api-access-2d7bz\") pod \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " Oct 31 01:28:40.690484 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.690456 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-openshift-ca\") pod \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\" (UID: \"578a76ef-aa57-4e07-ae0c-935ef3cc2243\") " Oct 31 01:28:40.690897 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.690872 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-openshift-ca" pod "578a76ef-aa57-4e07-ae0c-935ef3cc2243" (UID: "578a76ef-aa57-4e07-ae0c-935ef3cc2243"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:40.692465 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.692443 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-xks-ca" pod "578a76ef-aa57-4e07-ae0c-935ef3cc2243" (UID: "578a76ef-aa57-4e07-ae0c-935ef3cc2243"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:40.692567 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.692470 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578a76ef-aa57-4e07-ae0c-935ef3cc2243-kube-api-access-2d7bz" pod "578a76ef-aa57-4e07-ae0c-935ef3cc2243" (UID: "578a76ef-aa57-4e07-ae0c-935ef3cc2243"). InnerVolumeSpecName "kube-api-access-2d7bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:40.792332 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.792296 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d7bz\" (UniqueName: \"kubernetes.io/projected/578a76ef-aa57-4e07-ae0c-935ef3cc2243-kube-api-access-2d7bz\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:40.792332 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.792326 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:40.792493 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:40.792341 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/578a76ef-aa57-4e07-ae0c-935ef3cc2243-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:41.393831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:41.393799 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" Oct 31 01:28:41.394080 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:41.393840 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k" event={"ID":"578a76ef-aa57-4e07-ae0c-935ef3cc2243","Type":"ContainerDied","Data":"6cc3e7b012eda6e3a2dc8b10e2e690998efe8d75558259b01cb33e893dbec8ef"} Oct 31 01:28:41.394080 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:41.393893 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cc3e7b012eda6e3a2dc8b10e2e690998efe8d75558259b01cb33e893dbec8ef" Oct 31 01:28:41.503975 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:41.503946 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" podUID="9e30dbf8-db51-4772-9f60-852d38b8d182" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:28:41.503975 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:41.503959 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" podUID="61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:28:45.238074 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:45.238042 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx"] Oct 31 01:28:45.285187 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:45.285162 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9332a5-nkkqx"] Oct 31 01:28:46.609375 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:46.609345 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011e3744-fcb9-4729-8882-c8c42797b6dd" path="/var/lib/kubelet/pods/011e3744-fcb9-4729-8882-c8c42797b6dd/volumes" Oct 31 01:28:46.794370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:46.794338 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:28:46.897590 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:46.897566 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:28:46.953434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:46.953410 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e30dbf8-db51-4772-9f60-852d38b8d182-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e30dbf8-db51-4772-9f60-852d38b8d182-plugins-config-volume\") pod \"9e30dbf8-db51-4772-9f60-852d38b8d182\" (UID: \"9e30dbf8-db51-4772-9f60-852d38b8d182\") " Oct 31 01:28:46.953574 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:46.953441 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e30dbf8-db51-4772-9f60-852d38b8d182-kube-api-access-w7b4h\" (UniqueName: \"kubernetes.io/projected/9e30dbf8-db51-4772-9f60-852d38b8d182-kube-api-access-w7b4h\") pod \"9e30dbf8-db51-4772-9f60-852d38b8d182\" (UID: \"9e30dbf8-db51-4772-9f60-852d38b8d182\") " Oct 31 01:28:46.953855 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:46.953833 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e30dbf8-db51-4772-9f60-852d38b8d182-plugins-config-volume" pod "9e30dbf8-db51-4772-9f60-852d38b8d182" (UID: "9e30dbf8-db51-4772-9f60-852d38b8d182"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:46.955669 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:46.955643 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e30dbf8-db51-4772-9f60-852d38b8d182-kube-api-access-w7b4h" pod "9e30dbf8-db51-4772-9f60-852d38b8d182" (UID: "9e30dbf8-db51-4772-9f60-852d38b8d182"). InnerVolumeSpecName "kube-api-access-w7b4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:47.054830 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.054808 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-plugins-config-volume\") pod \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\" (UID: \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\") " Oct 31 01:28:47.054963 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.054874 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-kube-api-access-xbghm\" (UniqueName: \"kubernetes.io/projected/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-kube-api-access-xbghm\") pod \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\" (UID: \"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b\") " Oct 31 01:28:47.055247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.055222 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-plugins-config-volume" pod "61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" (UID: "61f4bbb5-318a-43d4-81e2-e907fb9b9b0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:47.055287 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.055260 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e30dbf8-db51-4772-9f60-852d38b8d182-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:47.055287 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.055276 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7b4h\" (UniqueName: \"kubernetes.io/projected/9e30dbf8-db51-4772-9f60-852d38b8d182-kube-api-access-w7b4h\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:47.057146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.057116 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-kube-api-access-xbghm" pod "61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" (UID: "61f4bbb5-318a-43d4-81e2-e907fb9b9b0b"). InnerVolumeSpecName "kube-api-access-xbghm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:47.156010 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.155947 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:47.156010 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.155970 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbghm\" (UniqueName: \"kubernetes.io/projected/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b-kube-api-access-xbghm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:47.434384 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.434315 5719 generic.go:356] "Generic (PLEG): container finished" podID="61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" containerID="95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386" exitCode=0 Oct 31 01:28:47.434384 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.434359 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" Oct 31 01:28:47.434570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.434394 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" event={"ID":"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b","Type":"ContainerDied","Data":"95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386"} Oct 31 01:28:47.434570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.434446 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s" event={"ID":"61f4bbb5-318a-43d4-81e2-e907fb9b9b0b","Type":"ContainerDied","Data":"09cd18a1e87d88326a887362035802f37ca449421f717e9ab7d57cba9e49ac85"} Oct 31 01:28:47.434570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.434475 5719 scope.go:122] "RemoveContainer" containerID="95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386" Oct 31 01:28:47.435740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.435717 5719 generic.go:356] "Generic (PLEG): container finished" podID="9e30dbf8-db51-4772-9f60-852d38b8d182" containerID="647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510" exitCode=0 Oct 31 01:28:47.435740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.435747 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" event={"ID":"9e30dbf8-db51-4772-9f60-852d38b8d182","Type":"ContainerDied","Data":"647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510"} Oct 31 01:28:47.435930 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.435779 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" event={"ID":"9e30dbf8-db51-4772-9f60-852d38b8d182","Type":"ContainerDied","Data":"781c7a0f1ae919743d855654da82ffeafeec245dd71b566f92ff29aab70eff52"} Oct 31 01:28:47.435930 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.435788 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf" Oct 31 01:28:47.445917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.445871 5719 scope.go:122] "RemoveContainer" containerID="95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386" Oct 31 01:28:47.446165 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:28:47.446146 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386\": container with ID starting with 95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386 not found: ID does not exist" containerID="95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386" Oct 31 01:28:47.446246 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.446179 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386"} err="failed to get container status \"95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386\": rpc error: code = NotFound desc = could not find container \"95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386\": container with ID starting with 95ed2545c7fd6294299c7d75da9217dfc3c53ddfaf98640b129d3f42b7e19386 not found: ID does not exist" Oct 31 01:28:47.446246 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.446201 5719 scope.go:122] "RemoveContainer" containerID="647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510" Oct 31 01:28:47.455278 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.455253 5719 scope.go:122] "RemoveContainer" containerID="647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510" Oct 31 01:28:47.455530 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:28:47.455504 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510\": container with ID starting with 647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510 not found: ID does not exist" containerID="647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510" Oct 31 01:28:47.455623 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.455540 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510"} err="failed to get container status \"647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510\": rpc error: code = NotFound desc = could not find container \"647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510\": container with ID starting with 647c2cf82b42931f625e0d674bdf4eef28f6af647a0cd7320062596a2ffd6510 not found: ID does not exist" Oct 31 01:28:47.594901 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.594870 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s"] Oct 31 01:28:47.641306 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.641281 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3be6fa-c4f89678c-9xl9s"] Oct 31 01:28:47.709841 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.707193 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf"] Oct 31 01:28:47.712785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:47.712761 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3be6fa-64ccb8dc67-p2zgf"] Oct 31 01:28:48.329911 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:48.329885 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt"] Oct 31 01:28:48.442346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:48.442315 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k"] Oct 31 01:28:48.602582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:48.602500 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" path="/var/lib/kubelet/pods/61f4bbb5-318a-43d4-81e2-e907fb9b9b0b/volumes" Oct 31 01:28:48.602970 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:48.602951 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e30dbf8-db51-4772-9f60-852d38b8d182" path="/var/lib/kubelet/pods/9e30dbf8-db51-4772-9f60-852d38b8d182/volumes" Oct 31 01:28:53.331082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.331042 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" podUID="0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" containerName="payload-processing" containerID="cri-o://123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c" gracePeriod=25 Oct 31 01:28:53.443309 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.443273 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" podUID="12f707cc-b82a-44a3-907d-d71336fa75db" containerName="payload-pre-processing" containerID="cri-o://248a060829df7164635d40e12ccc802ca57722bcbb9a926d2db57d6317478618" gracePeriod=25 Oct 31 01:28:53.473897 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.473870 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7"] Oct 31 01:28:53.474423 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474403 5719 cpu_manager.go:414] "containerMap: removing container" podUID="61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" containerName="payload-pre-processing" Oct 31 01:28:53.474423 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474421 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" containerName="payload-pre-processing" Oct 31 01:28:53.474569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474432 5719 cpu_manager.go:414] "containerMap: removing container" podUID="578a76ef-aa57-4e07-ae0c-935ef3cc2243" containerName="revoke-keys" Oct 31 01:28:53.474569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474439 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578a76ef-aa57-4e07-ae0c-935ef3cc2243" containerName="revoke-keys" Oct 31 01:28:53.474569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474450 5719 cpu_manager.go:414] "containerMap: removing container" podUID="9e30dbf8-db51-4772-9f60-852d38b8d182" containerName="payload-processing" Oct 31 01:28:53.474569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474455 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e30dbf8-db51-4772-9f60-852d38b8d182" containerName="payload-processing" Oct 31 01:28:53.474569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474529 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f4bbb5-318a-43d4-81e2-e907fb9b9b0b" containerName="payload-pre-processing" Oct 31 01:28:53.474569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474537 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="578a76ef-aa57-4e07-ae0c-935ef3cc2243" containerName="revoke-keys" Oct 31 01:28:53.474569 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.474543 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e30dbf8-db51-4772-9f60-852d38b8d182" containerName="payload-processing" Oct 31 01:28:53.479176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.479160 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.577639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.577614 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:28:53.577956 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.577940 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:53.578047 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.577944 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:28:53.578047 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.578031 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:28:53.580364 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.580334 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7"] Oct 31 01:28:53.616087 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.616016 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngc6t\" (UniqueName: \"kubernetes.io/projected/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-kube-api-access-ngc6t\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.616087 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.616051 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.616245 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.616139 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.717200 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.717177 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngc6t\" (UniqueName: \"kubernetes.io/projected/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-kube-api-access-ngc6t\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.717338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.717223 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.717338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.717283 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.717959 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.717941 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.719757 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.719736 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.776100 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.776081 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngc6t\" (UniqueName: \"kubernetes.io/projected/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-kube-api-access-ngc6t\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:53.792407 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:53.792386 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:54.164880 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:54.164848 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7"] Oct 31 01:28:54.172039 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:28:54.172012 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4eb967_4bb3_4be3_bd80_f78473c46e9f.slice/crio-a24b6a7d52ce7540589f48fdfcfdbd13d26a241808106fd85b4ee6d9a95b73ce WatchSource:0}: Error finding container a24b6a7d52ce7540589f48fdfcfdbd13d26a241808106fd85b4ee6d9a95b73ce: Status 404 returned error can't find the container with id a24b6a7d52ce7540589f48fdfcfdbd13d26a241808106fd85b4ee6d9a95b73ce Oct 31 01:28:54.482962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:54.482892 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" event={"ID":"6b4eb967-4bb3-4be3-bd80-f78473c46e9f","Type":"ContainerStarted","Data":"a24b6a7d52ce7540589f48fdfcfdbd13d26a241808106fd85b4ee6d9a95b73ce"} Oct 31 01:28:55.490819 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:55.490735 5719 generic.go:356] "Generic (PLEG): container finished" podID="6b4eb967-4bb3-4be3-bd80-f78473c46e9f" containerID="c30a352a0d734f332d643532790f3013eb72d5c35be85409a14b54e7145dda35" exitCode=0 Oct 31 01:28:55.491198 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:55.490819 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" event={"ID":"6b4eb967-4bb3-4be3-bd80-f78473c46e9f","Type":"ContainerDied","Data":"c30a352a0d734f332d643532790f3013eb72d5c35be85409a14b54e7145dda35"} Oct 31 01:28:56.639206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.639179 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:28:56.742936 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.742878 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-openshift-ca\") pod \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " Oct 31 01:28:56.742936 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.742913 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-xks-ca\") pod \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " Oct 31 01:28:56.743138 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.743007 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-kube-api-access-ngc6t\" (UniqueName: \"kubernetes.io/projected/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-kube-api-access-ngc6t\") pod \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\" (UID: \"6b4eb967-4bb3-4be3-bd80-f78473c46e9f\") " Oct 31 01:28:56.743288 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.743266 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-openshift-ca" pod "6b4eb967-4bb3-4be3-bd80-f78473c46e9f" (UID: "6b4eb967-4bb3-4be3-bd80-f78473c46e9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:56.745039 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.745020 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-xks-ca" pod "6b4eb967-4bb3-4be3-bd80-f78473c46e9f" (UID: "6b4eb967-4bb3-4be3-bd80-f78473c46e9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:56.745173 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.745155 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-kube-api-access-ngc6t" pod "6b4eb967-4bb3-4be3-bd80-f78473c46e9f" (UID: "6b4eb967-4bb3-4be3-bd80-f78473c46e9f"). InnerVolumeSpecName "kube-api-access-ngc6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:56.844656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.844625 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngc6t\" (UniqueName: \"kubernetes.io/projected/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-kube-api-access-ngc6t\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:56.844656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.844651 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:56.844832 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:56.844668 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4eb967-4bb3-4be3-bd80-f78473c46e9f-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:28:57.505078 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:57.505037 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" event={"ID":"6b4eb967-4bb3-4be3-bd80-f78473c46e9f","Type":"ContainerDied","Data":"a24b6a7d52ce7540589f48fdfcfdbd13d26a241808106fd85b4ee6d9a95b73ce"} Oct 31 01:28:57.505270 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:57.505092 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a24b6a7d52ce7540589f48fdfcfdbd13d26a241808106fd85b4ee6d9a95b73ce" Oct 31 01:28:57.505270 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:28:57.505052 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7" Oct 31 01:29:01.646379 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:01.646342 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" podUID="0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:29:01.646860 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:01.646342 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" podUID="12f707cc-b82a-44a3-907d-d71336fa75db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:29:03.010719 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:29:03.010659 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:29:11.645822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:11.645785 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" podUID="0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:29:11.646263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:11.645791 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" podUID="12f707cc-b82a-44a3-907d-d71336fa75db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:29:13.593464 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.593368 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:29:13.602306 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.602274 5719 generic.go:356] "Generic (PLEG): container finished" podID="12f707cc-b82a-44a3-907d-d71336fa75db" containerID="248a060829df7164635d40e12ccc802ca57722bcbb9a926d2db57d6317478618" exitCode=0 Oct 31 01:29:13.602430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.602338 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" event={"ID":"12f707cc-b82a-44a3-907d-d71336fa75db","Type":"ContainerDied","Data":"248a060829df7164635d40e12ccc802ca57722bcbb9a926d2db57d6317478618"} Oct 31 01:29:13.603718 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.603681 5719 generic.go:356] "Generic (PLEG): container finished" podID="0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" containerID="123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c" exitCode=0 Oct 31 01:29:13.603825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.603731 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" Oct 31 01:29:13.603825 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.603795 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" event={"ID":"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a","Type":"ContainerDied","Data":"123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c"} Oct 31 01:29:13.603900 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.603838 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt" event={"ID":"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a","Type":"ContainerDied","Data":"12c4fe65788be24496b69973e3aaa770d75aa732eb34f4cc3c117f4f5f3bc679"} Oct 31 01:29:13.603900 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.603868 5719 scope.go:122] "RemoveContainer" containerID="123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c" Oct 31 01:29:13.613936 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.613873 5719 scope.go:122] "RemoveContainer" containerID="123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c" Oct 31 01:29:13.614153 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:29:13.614135 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c\": container with ID starting with 123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c not found: ID does not exist" containerID="123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c" Oct 31 01:29:13.614222 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.614166 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c"} err="failed to get container status \"123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c\": rpc error: code = NotFound desc = could not find container \"123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c\": container with ID starting with 123bd9101f1f631421a883df6d28fdf46dd0d9edced87f800ed93f645018352c not found: ID does not exist" Oct 31 01:29:13.696605 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.696583 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:29:13.700534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.700517 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-plugins-config-volume\") pod \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\" (UID: \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\") " Oct 31 01:29:13.700648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.700635 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-kube-api-access-pz6fv\" (UniqueName: \"kubernetes.io/projected/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-kube-api-access-pz6fv\") pod \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\" (UID: \"0186e230-3fc0-4c2a-b9c3-18da6fa83a1a\") " Oct 31 01:29:13.700896 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.700875 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-plugins-config-volume" pod "0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" (UID: "0186e230-3fc0-4c2a-b9c3-18da6fa83a1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:13.702729 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.702708 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-kube-api-access-pz6fv" pod "0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" (UID: "0186e230-3fc0-4c2a-b9c3-18da6fa83a1a"). InnerVolumeSpecName "kube-api-access-pz6fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:13.801655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.801635 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f707cc-b82a-44a3-907d-d71336fa75db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f707cc-b82a-44a3-907d-d71336fa75db-plugins-config-volume\") pod \"12f707cc-b82a-44a3-907d-d71336fa75db\" (UID: \"12f707cc-b82a-44a3-907d-d71336fa75db\") " Oct 31 01:29:13.801799 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.801777 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f707cc-b82a-44a3-907d-d71336fa75db-kube-api-access-gxlwd\" (UniqueName: \"kubernetes.io/projected/12f707cc-b82a-44a3-907d-d71336fa75db-kube-api-access-gxlwd\") pod \"12f707cc-b82a-44a3-907d-d71336fa75db\" (UID: \"12f707cc-b82a-44a3-907d-d71336fa75db\") " Oct 31 01:29:13.802019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.801999 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f707cc-b82a-44a3-907d-d71336fa75db-plugins-config-volume" pod "12f707cc-b82a-44a3-907d-d71336fa75db" (UID: "12f707cc-b82a-44a3-907d-d71336fa75db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:13.802019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.802016 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:13.802109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.802027 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz6fv\" (UniqueName: \"kubernetes.io/projected/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a-kube-api-access-pz6fv\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:13.803884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.803865 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f707cc-b82a-44a3-907d-d71336fa75db-kube-api-access-gxlwd" pod "12f707cc-b82a-44a3-907d-d71336fa75db" (UID: "12f707cc-b82a-44a3-907d-d71336fa75db"). InnerVolumeSpecName "kube-api-access-gxlwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:13.903142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.903080 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxlwd\" (UniqueName: \"kubernetes.io/projected/12f707cc-b82a-44a3-907d-d71336fa75db-kube-api-access-gxlwd\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:13.903142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:13.903106 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12f707cc-b82a-44a3-907d-d71336fa75db-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:14.019988 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:14.019962 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt"] Oct 31 01:29:14.023779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:14.023755 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f9925a-bd8fbb5dc-brvjt"] Oct 31 01:29:14.602410 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:14.602343 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" path="/var/lib/kubelet/pods/0186e230-3fc0-4c2a-b9c3-18da6fa83a1a/volumes" Oct 31 01:29:14.610958 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:14.610927 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" event={"ID":"12f707cc-b82a-44a3-907d-d71336fa75db","Type":"ContainerDied","Data":"49201749d2202f17a33284a5451abb594709c0edbe4d004be6ff900d4e4bd0c3"} Oct 31 01:29:14.611075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:14.610959 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k" Oct 31 01:29:14.611075 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:14.610989 5719 scope.go:122] "RemoveContainer" containerID="248a060829df7164635d40e12ccc802ca57722bcbb9a926d2db57d6317478618" Oct 31 01:29:14.770764 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:14.770737 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k"] Oct 31 01:29:14.774954 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:14.774933 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f9925a-d9db6c895-hzj5k"] Oct 31 01:29:16.602201 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:16.602162 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f707cc-b82a-44a3-907d-d71336fa75db" path="/var/lib/kubelet/pods/12f707cc-b82a-44a3-907d-d71336fa75db/volumes" Oct 31 01:29:17.085606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:17.085574 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:29:21.888180 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:21.888147 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct"] Oct 31 01:29:21.935795 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:21.935760 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3be6fa-dtlct"] Oct 31 01:29:22.602752 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:22.602679 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856d1257-dc8a-47d1-a61c-66b8f1615366" path="/var/lib/kubelet/pods/856d1257-dc8a-47d1-a61c-66b8f1615366/volumes" Oct 31 01:29:27.929635 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:27.929599 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p"] Oct 31 01:29:27.977237 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:27.977201 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-dd98f0-5tb5p"] Oct 31 01:29:28.603914 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:28.603884 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e66432f-688b-4aa6-b770-6391ab662009" path="/var/lib/kubelet/pods/0e66432f-688b-4aa6-b770-6391ab662009/volumes" Oct 31 01:29:29.112801 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:29.112768 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k"] Oct 31 01:29:29.160456 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:29.160429 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f9925a-qhr2k"] Oct 31 01:29:30.464761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.464731 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd"] Oct 31 01:29:30.465382 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465361 5719 cpu_manager.go:414] "containerMap: removing container" podUID="0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" containerName="payload-processing" Oct 31 01:29:30.465486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465383 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" containerName="payload-processing" Oct 31 01:29:30.465486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465404 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6b4eb967-4bb3-4be3-bd80-f78473c46e9f" containerName="revoke-keys" Oct 31 01:29:30.465486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465412 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4eb967-4bb3-4be3-bd80-f78473c46e9f" containerName="revoke-keys" Oct 31 01:29:30.465486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465445 5719 cpu_manager.go:414] "containerMap: removing container" podUID="12f707cc-b82a-44a3-907d-d71336fa75db" containerName="payload-pre-processing" Oct 31 01:29:30.465486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465453 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f707cc-b82a-44a3-907d-d71336fa75db" containerName="payload-pre-processing" Oct 31 01:29:30.465782 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465600 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="0186e230-3fc0-4c2a-b9c3-18da6fa83a1a" containerName="payload-processing" Oct 31 01:29:30.465782 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465614 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4eb967-4bb3-4be3-bd80-f78473c46e9f" containerName="revoke-keys" Oct 31 01:29:30.465782 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.465631 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f707cc-b82a-44a3-907d-d71336fa75db" containerName="payload-pre-processing" Oct 31 01:29:30.470512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.470491 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.520835 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.520816 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:29:30.567604 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.567583 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmvmx\" (UniqueName: \"kubernetes.io/projected/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-kube-api-access-hmvmx\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.567733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.567638 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.567794 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.567739 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.568395 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.568377 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:29:30.568466 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.568421 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:29:30.568514 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.568485 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:29:30.570914 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.570898 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd"] Oct 31 01:29:30.610206 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.610178 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578a76ef-aa57-4e07-ae0c-935ef3cc2243" path="/var/lib/kubelet/pods/578a76ef-aa57-4e07-ae0c-935ef3cc2243/volumes" Oct 31 01:29:30.669040 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.669011 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmvmx\" (UniqueName: \"kubernetes.io/projected/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-kube-api-access-hmvmx\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.669155 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.669056 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.669155 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.669089 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.669752 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.669734 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.671492 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.671466 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.728667 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.728612 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmvmx\" (UniqueName: \"kubernetes.io/projected/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-kube-api-access-hmvmx\") pod \"maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:30.785706 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:30.785655 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:31.158619 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:31.158585 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd"] Oct 31 01:29:31.166410 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:29:31.166380 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc6c5cf_8749_49b4_88dd_6dd6a25c6845.slice/crio-127b11e16e1330c8695f239a1871546db15b6f91e645a08e5f6d0fe23d890852 WatchSource:0}: Error finding container 127b11e16e1330c8695f239a1871546db15b6f91e645a08e5f6d0fe23d890852: Status 404 returned error can't find the container with id 127b11e16e1330c8695f239a1871546db15b6f91e645a08e5f6d0fe23d890852 Oct 31 01:29:31.710317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:31.710244 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" event={"ID":"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845","Type":"ContainerStarted","Data":"e1c13d74a695d3ca6548f38da4c229b4e587b5b41940af92fc5e967a2ef38226"} Oct 31 01:29:31.710317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:31.710307 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" event={"ID":"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845","Type":"ContainerStarted","Data":"127b11e16e1330c8695f239a1871546db15b6f91e645a08e5f6d0fe23d890852"} Oct 31 01:29:31.822013 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:31.821967 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" podStartSLOduration=1.821954166 podStartE2EDuration="1.821954166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:31.820880556 +0000 UTC m=+3653.357022858" watchObservedRunningTime="2026-10-31 01:29:31.821954166 +0000 UTC m=+3653.358096486" Oct 31 01:29:32.717113 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:32.717079 5719 generic.go:356] "Generic (PLEG): container finished" podID="cfc6c5cf-8749-49b4-88dd-6dd6a25c6845" containerID="e1c13d74a695d3ca6548f38da4c229b4e587b5b41940af92fc5e967a2ef38226" exitCode=0 Oct 31 01:29:32.717505 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:32.717166 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" event={"ID":"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845","Type":"ContainerDied","Data":"e1c13d74a695d3ca6548f38da4c229b4e587b5b41940af92fc5e967a2ef38226"} Oct 31 01:29:33.009701 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:29:33.009614 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:29:33.866701 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:33.866673 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:33.901375 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:33.901347 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-openshift-ca\") pod \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " Oct 31 01:29:33.901501 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:33.901432 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-kube-api-access-hmvmx\" (UniqueName: \"kubernetes.io/projected/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-kube-api-access-hmvmx\") pod \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " Oct 31 01:29:33.901573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:33.901512 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-xks-ca\") pod \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\" (UID: \"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845\") " Oct 31 01:29:33.901810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:33.901776 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-openshift-ca" pod "cfc6c5cf-8749-49b4-88dd-6dd6a25c6845" (UID: "cfc6c5cf-8749-49b4-88dd-6dd6a25c6845"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:33.902035 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:33.902016 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:33.903869 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:33.903831 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-kube-api-access-hmvmx" pod "cfc6c5cf-8749-49b4-88dd-6dd6a25c6845" (UID: "cfc6c5cf-8749-49b4-88dd-6dd6a25c6845"). InnerVolumeSpecName "kube-api-access-hmvmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:33.903971 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:33.903898 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-xks-ca" pod "cfc6c5cf-8749-49b4-88dd-6dd6a25c6845" (UID: "cfc6c5cf-8749-49b4-88dd-6dd6a25c6845"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:34.002997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:34.002922 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmvmx\" (UniqueName: \"kubernetes.io/projected/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-kube-api-access-hmvmx\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.002997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:34.002951 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:34.730276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:34.730245 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" event={"ID":"cfc6c5cf-8749-49b4-88dd-6dd6a25c6845","Type":"ContainerDied","Data":"127b11e16e1330c8695f239a1871546db15b6f91e645a08e5f6d0fe23d890852"} Oct 31 01:29:34.730276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:34.730273 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd" Oct 31 01:29:34.730464 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:34.730290 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="127b11e16e1330c8695f239a1871546db15b6f91e645a08e5f6d0fe23d890852" Oct 31 01:29:34.969220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:34.969139 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz"] Oct 31 01:29:35.601134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:35.601107 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7"] Oct 31 01:29:35.604620 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:35.604594 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-8fb7f0-gl2p7"] Oct 31 01:29:36.602556 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:36.602524 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4eb967-4bb3-4be3-bd80-f78473c46e9f" path="/var/lib/kubelet/pods/6b4eb967-4bb3-4be3-bd80-f78473c46e9f/volumes" Oct 31 01:29:38.389503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.389472 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6"] Oct 31 01:29:38.389961 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.389946 5719 cpu_manager.go:414] "containerMap: removing container" podUID="cfc6c5cf-8749-49b4-88dd-6dd6a25c6845" containerName="revoke-keys" Oct 31 01:29:38.390007 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.389963 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc6c5cf-8749-49b4-88dd-6dd6a25c6845" containerName="revoke-keys" Oct 31 01:29:38.390079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.390070 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc6c5cf-8749-49b4-88dd-6dd6a25c6845" containerName="revoke-keys" Oct 31 01:29:38.393546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.393527 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.449507 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.449482 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.449657 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.449550 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.449787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.449722 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7rs8\" (UniqueName: \"kubernetes.io/projected/1f702c89-e58f-45ad-91b4-710cbb4cf351-kube-api-access-b7rs8\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.492193 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.492160 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:29:38.492193 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.492172 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:29:38.492391 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.492244 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:29:38.492391 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.492247 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:29:38.494345 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.494324 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6"] Oct 31 01:29:38.551193 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.551169 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.551329 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.551205 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.551329 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.551244 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rs8\" (UniqueName: \"kubernetes.io/projected/1f702c89-e58f-45ad-91b4-710cbb4cf351-kube-api-access-b7rs8\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.551867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.551846 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.553723 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.553710 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.656737 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.656666 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:29:38.660016 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.659991 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rs8\" (UniqueName: \"kubernetes.io/projected/1f702c89-e58f-45ad-91b4-710cbb4cf351-kube-api-access-b7rs8\") pod \"maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:38.755238 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.755214 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:29:38.756653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:38.756632 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:39.145824 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:39.145796 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6"] Oct 31 01:29:39.147088 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:29:39.147056 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f702c89_e58f_45ad_91b4_710cbb4cf351.slice/crio-58a11af8d8189e3bbdb450156a1f38c5c6ca272f3e04114a79147f1e94278cb0 WatchSource:0}: Error finding container 58a11af8d8189e3bbdb450156a1f38c5c6ca272f3e04114a79147f1e94278cb0: Status 404 returned error can't find the container with id 58a11af8d8189e3bbdb450156a1f38c5c6ca272f3e04114a79147f1e94278cb0 Oct 31 01:29:39.766879 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:39.766796 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" event={"ID":"1f702c89-e58f-45ad-91b4-710cbb4cf351","Type":"ContainerStarted","Data":"3fb2b54cb846229f12233887d4eb55f815de7507d709a5ea7fa347be6ae3c803"} Oct 31 01:29:39.767301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:39.766896 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" event={"ID":"1f702c89-e58f-45ad-91b4-710cbb4cf351","Type":"ContainerStarted","Data":"58a11af8d8189e3bbdb450156a1f38c5c6ca272f3e04114a79147f1e94278cb0"} Oct 31 01:29:39.897995 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:39.897948 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" podStartSLOduration=1.897935932 podStartE2EDuration="1.897935932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:39.897418122 +0000 UTC m=+3661.433560428" watchObservedRunningTime="2026-10-31 01:29:39.897935932 +0000 UTC m=+3661.434078234" Oct 31 01:29:40.774553 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:40.774517 5719 generic.go:356] "Generic (PLEG): container finished" podID="1f702c89-e58f-45ad-91b4-710cbb4cf351" containerID="3fb2b54cb846229f12233887d4eb55f815de7507d709a5ea7fa347be6ae3c803" exitCode=0 Oct 31 01:29:40.774977 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:40.774607 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" event={"ID":"1f702c89-e58f-45ad-91b4-710cbb4cf351","Type":"ContainerDied","Data":"3fb2b54cb846229f12233887d4eb55f815de7507d709a5ea7fa347be6ae3c803"} Oct 31 01:29:41.928487 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:41.928459 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:41.985158 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:41.985132 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-openshift-ca\") pod \"1f702c89-e58f-45ad-91b4-710cbb4cf351\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " Oct 31 01:29:41.985279 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:41.985207 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f702c89-e58f-45ad-91b4-710cbb4cf351-kube-api-access-b7rs8\" (UniqueName: \"kubernetes.io/projected/1f702c89-e58f-45ad-91b4-710cbb4cf351-kube-api-access-b7rs8\") pod \"1f702c89-e58f-45ad-91b4-710cbb4cf351\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " Oct 31 01:29:41.985279 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:41.985244 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-xks-ca\") pod \"1f702c89-e58f-45ad-91b4-710cbb4cf351\" (UID: \"1f702c89-e58f-45ad-91b4-710cbb4cf351\") " Oct 31 01:29:41.985434 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:41.985413 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-openshift-ca" pod "1f702c89-e58f-45ad-91b4-710cbb4cf351" (UID: "1f702c89-e58f-45ad-91b4-710cbb4cf351"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:41.985708 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:41.985674 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:41.987318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:41.987298 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f702c89-e58f-45ad-91b4-710cbb4cf351-kube-api-access-b7rs8" pod "1f702c89-e58f-45ad-91b4-710cbb4cf351" (UID: "1f702c89-e58f-45ad-91b4-710cbb4cf351"). InnerVolumeSpecName "kube-api-access-b7rs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:41.987429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:41.987355 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-xks-ca" pod "1f702c89-e58f-45ad-91b4-710cbb4cf351" (UID: "1f702c89-e58f-45ad-91b4-710cbb4cf351"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:42.086927 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:42.086897 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7rs8\" (UniqueName: \"kubernetes.io/projected/1f702c89-e58f-45ad-91b4-710cbb4cf351-kube-api-access-b7rs8\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:42.087094 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:42.086933 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f702c89-e58f-45ad-91b4-710cbb4cf351-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:42.787130 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:42.787103 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" Oct 31 01:29:42.787266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:42.787134 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6" event={"ID":"1f702c89-e58f-45ad-91b4-710cbb4cf351","Type":"ContainerDied","Data":"58a11af8d8189e3bbdb450156a1f38c5c6ca272f3e04114a79147f1e94278cb0"} Oct 31 01:29:42.787266 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:42.787176 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a11af8d8189e3bbdb450156a1f38c5c6ca272f3e04114a79147f1e94278cb0" Oct 31 01:29:42.891324 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:42.891247 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26"] Oct 31 01:29:42.893677 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:42.893555 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj"] Oct 31 01:29:45.221115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.221077 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57"] Oct 31 01:29:45.221577 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.221565 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1f702c89-e58f-45ad-91b4-710cbb4cf351" containerName="revoke-keys" Oct 31 01:29:45.221623 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.221579 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f702c89-e58f-45ad-91b4-710cbb4cf351" containerName="revoke-keys" Oct 31 01:29:45.221676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.221665 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f702c89-e58f-45ad-91b4-710cbb4cf351" containerName="revoke-keys" Oct 31 01:29:45.226218 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.226196 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.277826 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.277620 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:29:45.280187 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.280155 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57"] Oct 31 01:29:45.321603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.321572 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.321740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.321644 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nnkr\" (UniqueName: \"kubernetes.io/projected/1c3e65eb-259c-44d6-8711-be9fc63744f0-kube-api-access-8nnkr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.321792 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.321762 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.325167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.325148 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:29:45.325233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.325148 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:29:45.325292 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.325281 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:29:45.423170 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.423146 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.423276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.423212 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nnkr\" (UniqueName: \"kubernetes.io/projected/1c3e65eb-259c-44d6-8711-be9fc63744f0-kube-api-access-8nnkr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.423276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.423243 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.423894 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.423877 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.425789 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.425765 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.482370 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.482319 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nnkr\" (UniqueName: \"kubernetes.io/projected/1c3e65eb-259c-44d6-8711-be9fc63744f0-kube-api-access-8nnkr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.539108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.539086 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:45.912838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:45.912804 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57"] Oct 31 01:29:45.920041 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:29:45.920011 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3e65eb_259c_44d6_8711_be9fc63744f0.slice/crio-9e3492659e27ca955312a82c81d2ae611d57eb530aa6367861d5ddbd7f9c1e2d WatchSource:0}: Error finding container 9e3492659e27ca955312a82c81d2ae611d57eb530aa6367861d5ddbd7f9c1e2d: Status 404 returned error can't find the container with id 9e3492659e27ca955312a82c81d2ae611d57eb530aa6367861d5ddbd7f9c1e2d Oct 31 01:29:46.811344 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:46.811313 5719 generic.go:356] "Generic (PLEG): container finished" podID="1c3e65eb-259c-44d6-8711-be9fc63744f0" containerID="5b8d6bc0a82e2275337edccd4e1c6e159f01efe7f1221372f29906f7f58394bc" exitCode=0 Oct 31 01:29:46.811673 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:46.811455 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" event={"ID":"1c3e65eb-259c-44d6-8711-be9fc63744f0","Type":"ContainerDied","Data":"5b8d6bc0a82e2275337edccd4e1c6e159f01efe7f1221372f29906f7f58394bc"} Oct 31 01:29:46.811673 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:46.811505 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" event={"ID":"1c3e65eb-259c-44d6-8711-be9fc63744f0","Type":"ContainerStarted","Data":"9e3492659e27ca955312a82c81d2ae611d57eb530aa6367861d5ddbd7f9c1e2d"} Oct 31 01:29:47.962247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:47.962216 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:48.049930 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.049890 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-openshift-ca\") pod \"1c3e65eb-259c-44d6-8711-be9fc63744f0\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " Oct 31 01:29:48.050111 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.049991 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3e65eb-259c-44d6-8711-be9fc63744f0-kube-api-access-8nnkr\" (UniqueName: \"kubernetes.io/projected/1c3e65eb-259c-44d6-8711-be9fc63744f0-kube-api-access-8nnkr\") pod \"1c3e65eb-259c-44d6-8711-be9fc63744f0\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " Oct 31 01:29:48.050111 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.050052 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-xks-ca\") pod \"1c3e65eb-259c-44d6-8711-be9fc63744f0\" (UID: \"1c3e65eb-259c-44d6-8711-be9fc63744f0\") " Oct 31 01:29:48.050331 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.050238 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-openshift-ca" pod "1c3e65eb-259c-44d6-8711-be9fc63744f0" (UID: "1c3e65eb-259c-44d6-8711-be9fc63744f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:48.050549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.050524 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:48.052267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.052247 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3e65eb-259c-44d6-8711-be9fc63744f0-kube-api-access-8nnkr" pod "1c3e65eb-259c-44d6-8711-be9fc63744f0" (UID: "1c3e65eb-259c-44d6-8711-be9fc63744f0"). InnerVolumeSpecName "kube-api-access-8nnkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:48.052394 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.052375 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-xks-ca" pod "1c3e65eb-259c-44d6-8711-be9fc63744f0" (UID: "1c3e65eb-259c-44d6-8711-be9fc63744f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:48.152078 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.152011 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nnkr\" (UniqueName: \"kubernetes.io/projected/1c3e65eb-259c-44d6-8711-be9fc63744f0-kube-api-access-8nnkr\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:48.152078 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.152035 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c3e65eb-259c-44d6-8711-be9fc63744f0-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:48.824475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.824441 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" Oct 31 01:29:48.824642 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.824480 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57" event={"ID":"1c3e65eb-259c-44d6-8711-be9fc63744f0","Type":"ContainerDied","Data":"9e3492659e27ca955312a82c81d2ae611d57eb530aa6367861d5ddbd7f9c1e2d"} Oct 31 01:29:48.824642 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:48.824532 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e3492659e27ca955312a82c81d2ae611d57eb530aa6367861d5ddbd7f9c1e2d" Oct 31 01:29:49.991192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:49.991159 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" containerID="cri-o://0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd" gracePeriod=45 Oct 31 01:29:50.252885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.252823 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 01:29:50.371763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.371737 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-model-cache\") pod \"45a01aef-4f87-4f99-8a1e-b022e707afdb\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " Oct 31 01:29:50.371907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.371796 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-home\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-home\") pod \"45a01aef-4f87-4f99-8a1e-b022e707afdb\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " Oct 31 01:29:50.371907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.371843 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a01aef-4f87-4f99-8a1e-b022e707afdb-kube-api-access-r9tdf\" (UniqueName: \"kubernetes.io/projected/45a01aef-4f87-4f99-8a1e-b022e707afdb-kube-api-access-r9tdf\") pod \"45a01aef-4f87-4f99-8a1e-b022e707afdb\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " Oct 31 01:29:50.371907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.371895 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a01aef-4f87-4f99-8a1e-b022e707afdb-tls-certs\" (UniqueName: \"kubernetes.io/secret/45a01aef-4f87-4f99-8a1e-b022e707afdb-tls-certs\") pod \"45a01aef-4f87-4f99-8a1e-b022e707afdb\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " Oct 31 01:29:50.372100 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.371999 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-dshm\") pod \"45a01aef-4f87-4f99-8a1e-b022e707afdb\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " Oct 31 01:29:50.372100 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.372044 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-tmp-dir\") pod \"45a01aef-4f87-4f99-8a1e-b022e707afdb\" (UID: \"45a01aef-4f87-4f99-8a1e-b022e707afdb\") " Oct 31 01:29:50.372479 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.372034 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-model-cache" pod "45a01aef-4f87-4f99-8a1e-b022e707afdb" (UID: "45a01aef-4f87-4f99-8a1e-b022e707afdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.372664 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.372085 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-home" pod "45a01aef-4f87-4f99-8a1e-b022e707afdb" (UID: "45a01aef-4f87-4f99-8a1e-b022e707afdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.373005 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.372510 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-tmp-dir" pod "45a01aef-4f87-4f99-8a1e-b022e707afdb" (UID: "45a01aef-4f87-4f99-8a1e-b022e707afdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.374422 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.374396 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-dshm" pod "45a01aef-4f87-4f99-8a1e-b022e707afdb" (UID: "45a01aef-4f87-4f99-8a1e-b022e707afdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:50.374550 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.374534 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a01aef-4f87-4f99-8a1e-b022e707afdb-kube-api-access-r9tdf" pod "45a01aef-4f87-4f99-8a1e-b022e707afdb" (UID: "45a01aef-4f87-4f99-8a1e-b022e707afdb"). InnerVolumeSpecName "kube-api-access-r9tdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:50.374609 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.374556 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a01aef-4f87-4f99-8a1e-b022e707afdb-tls-certs" pod "45a01aef-4f87-4f99-8a1e-b022e707afdb" (UID: "45a01aef-4f87-4f99-8a1e-b022e707afdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:50.473741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.473683 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.473741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.473731 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9tdf\" (UniqueName: \"kubernetes.io/projected/45a01aef-4f87-4f99-8a1e-b022e707afdb-kube-api-access-r9tdf\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.473741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.473740 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a01aef-4f87-4f99-8a1e-b022e707afdb-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.473741 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.473748 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.473993 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.473773 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.473993 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.473782 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a01aef-4f87-4f99-8a1e-b022e707afdb-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:50.836568 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.836538 5719 generic.go:356] "Generic (PLEG): container finished" podID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerID="0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd" exitCode=0 Oct 31 01:29:50.836755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.836583 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" event={"ID":"45a01aef-4f87-4f99-8a1e-b022e707afdb","Type":"ContainerDied","Data":"0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd"} Oct 31 01:29:50.836755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.836594 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" Oct 31 01:29:50.836755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.836609 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz" event={"ID":"45a01aef-4f87-4f99-8a1e-b022e707afdb","Type":"ContainerDied","Data":"b6305957ee83f52bbb6b40aa1c32cc437f3ecc12359286852cda96b45ae4eb66"} Oct 31 01:29:50.836755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.836624 5719 scope.go:122] "RemoveContainer" containerID="0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd" Oct 31 01:29:50.845899 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.845873 5719 scope.go:122] "RemoveContainer" containerID="0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd" Oct 31 01:29:50.846150 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:29:50.846129 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd\": container with ID starting with 0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd not found: ID does not exist" containerID="0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd" Oct 31 01:29:50.846218 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.846162 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd"} err="failed to get container status \"0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd\": rpc error: code = NotFound desc = could not find container \"0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd\": container with ID starting with 0c3966fda7fd32c1818510c137cb7e393aab0b399a8f1bc142350435e538f0dd not found: ID does not exist" Oct 31 01:29:50.994220 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:50.994152 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz"] Oct 31 01:29:51.041570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:51.041547 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b63c24/simulator-w1-b63c24-kserve-66f96bf76d-vwccz"] Oct 31 01:29:52.603588 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:52.603543 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" path="/var/lib/kubelet/pods/45a01aef-4f87-4f99-8a1e-b022e707afdb/volumes" Oct 31 01:29:56.763317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:56.763275 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782"] Oct 31 01:29:56.871856 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:56.871827 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l"] Oct 31 01:29:57.913117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:57.913058 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" containerID="cri-o://3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd" gracePeriod=45 Oct 31 01:29:57.914470 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:57.914437 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" containerID="cri-o://d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6" gracePeriod=45 Oct 31 01:29:58.173540 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.173476 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 01:29:58.201625 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.201603 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 01:29:58.250076 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250048 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-model-cache\") pod \"3c759e29-fdad-44b7-a89f-755060c2c483\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " Oct 31 01:29:58.250240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250131 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c759e29-fdad-44b7-a89f-755060c2c483-kube-api-access-h9bw4\" (UniqueName: \"kubernetes.io/projected/3c759e29-fdad-44b7-a89f-755060c2c483-kube-api-access-h9bw4\") pod \"3c759e29-fdad-44b7-a89f-755060c2c483\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " Oct 31 01:29:58.250240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250165 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-home\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-home\") pod \"3c759e29-fdad-44b7-a89f-755060c2c483\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " Oct 31 01:29:58.250240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250219 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c759e29-fdad-44b7-a89f-755060c2c483-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c759e29-fdad-44b7-a89f-755060c2c483-tls-certs\") pod \"3c759e29-fdad-44b7-a89f-755060c2c483\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " Oct 31 01:29:58.250429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250259 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-dshm\") pod \"3c759e29-fdad-44b7-a89f-755060c2c483\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " Oct 31 01:29:58.250429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250314 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-tmp-dir\") pod \"3c759e29-fdad-44b7-a89f-755060c2c483\" (UID: \"3c759e29-fdad-44b7-a89f-755060c2c483\") " Oct 31 01:29:58.250429 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250350 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-model-cache" pod "3c759e29-fdad-44b7-a89f-755060c2c483" (UID: "3c759e29-fdad-44b7-a89f-755060c2c483"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.250804 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250786 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.250891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.250824 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-tmp-dir" pod "3c759e29-fdad-44b7-a89f-755060c2c483" (UID: "3c759e29-fdad-44b7-a89f-755060c2c483"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.251454 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.251432 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-home" pod "3c759e29-fdad-44b7-a89f-755060c2c483" (UID: "3c759e29-fdad-44b7-a89f-755060c2c483"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.253141 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.253113 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-dshm" pod "3c759e29-fdad-44b7-a89f-755060c2c483" (UID: "3c759e29-fdad-44b7-a89f-755060c2c483"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.253244 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.253120 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c759e29-fdad-44b7-a89f-755060c2c483-kube-api-access-h9bw4" pod "3c759e29-fdad-44b7-a89f-755060c2c483" (UID: "3c759e29-fdad-44b7-a89f-755060c2c483"). InnerVolumeSpecName "kube-api-access-h9bw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:58.253374 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.253351 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c759e29-fdad-44b7-a89f-755060c2c483-tls-certs" pod "3c759e29-fdad-44b7-a89f-755060c2c483" (UID: "3c759e29-fdad-44b7-a89f-755060c2c483"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:58.261483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.261462 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:29:58.352144 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352117 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-dshm\") pod \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " Oct 31 01:29:58.352291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352165 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa568cd6-7a3e-4f31-8271-781a4f9d4318-kube-api-access-lbfqr\" (UniqueName: \"kubernetes.io/projected/fa568cd6-7a3e-4f31-8271-781a4f9d4318-kube-api-access-lbfqr\") pod \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " Oct 31 01:29:58.352291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352246 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-home\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-home\") pod \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " Oct 31 01:29:58.352291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352279 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-model-cache\") pod \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " Oct 31 01:29:58.352448 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352424 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tls-certs\") pod \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " Oct 31 01:29:58.352655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352460 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tmp-dir\") pod \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\" (UID: \"fa568cd6-7a3e-4f31-8271-781a4f9d4318\") " Oct 31 01:29:58.352655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352570 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-home" pod "fa568cd6-7a3e-4f31-8271-781a4f9d4318" (UID: "fa568cd6-7a3e-4f31-8271-781a4f9d4318"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.352841 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352643 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-model-cache" pod "fa568cd6-7a3e-4f31-8271-781a4f9d4318" (UID: "fa568cd6-7a3e-4f31-8271-781a4f9d4318"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.353009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.352987 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9bw4\" (UniqueName: \"kubernetes.io/projected/3c759e29-fdad-44b7-a89f-755060c2c483-kube-api-access-h9bw4\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.353012 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.353026 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c759e29-fdad-44b7-a89f-755060c2c483-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.353027 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tmp-dir" pod "fa568cd6-7a3e-4f31-8271-781a4f9d4318" (UID: "fa568cd6-7a3e-4f31-8271-781a4f9d4318"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.353163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.353042 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.353051 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c759e29-fdad-44b7-a89f-755060c2c483-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.353058 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.353163 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.353066 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.354450 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.354423 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-dshm" pod "fa568cd6-7a3e-4f31-8271-781a4f9d4318" (UID: "fa568cd6-7a3e-4f31-8271-781a4f9d4318"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:58.355164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.355139 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tls-certs" pod "fa568cd6-7a3e-4f31-8271-781a4f9d4318" (UID: "fa568cd6-7a3e-4f31-8271-781a4f9d4318"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:58.355164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.355151 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa568cd6-7a3e-4f31-8271-781a4f9d4318-kube-api-access-lbfqr" pod "fa568cd6-7a3e-4f31-8271-781a4f9d4318" (UID: "fa568cd6-7a3e-4f31-8271-781a4f9d4318"). InnerVolumeSpecName "kube-api-access-lbfqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:58.454318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.454251 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.454318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.454280 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.454318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.454288 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa568cd6-7a3e-4f31-8271-781a4f9d4318-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.454318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.454298 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfqr\" (UniqueName: \"kubernetes.io/projected/fa568cd6-7a3e-4f31-8271-781a4f9d4318-kube-api-access-lbfqr\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:29:58.887221 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.887191 5719 generic.go:356] "Generic (PLEG): container finished" podID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerID="d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6" exitCode=0 Oct 31 01:29:58.887394 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.887229 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" event={"ID":"fa568cd6-7a3e-4f31-8271-781a4f9d4318","Type":"ContainerDied","Data":"d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6"} Oct 31 01:29:58.887394 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.887261 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" Oct 31 01:29:58.887394 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.887284 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj" event={"ID":"fa568cd6-7a3e-4f31-8271-781a4f9d4318","Type":"ContainerDied","Data":"97991509ea7a9923721d767d29dcafe168476f153be43999dd68d2bf99c4d364"} Oct 31 01:29:58.887394 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.887302 5719 scope.go:122] "RemoveContainer" containerID="d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6" Oct 31 01:29:58.889083 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.889059 5719 generic.go:356] "Generic (PLEG): container finished" podID="3c759e29-fdad-44b7-a89f-755060c2c483" containerID="3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd" exitCode=0 Oct 31 01:29:58.889227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.889111 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" Oct 31 01:29:58.889227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.889164 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" event={"ID":"3c759e29-fdad-44b7-a89f-755060c2c483","Type":"ContainerDied","Data":"3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd"} Oct 31 01:29:58.889227 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.889200 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" event={"ID":"3c759e29-fdad-44b7-a89f-755060c2c483","Type":"ContainerDied","Data":"d92db507175e49d358a3612f70aeeaadc0f1351c2b9e67e484a5177ac144d169"} Oct 31 01:29:58.900607 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.900588 5719 scope.go:122] "RemoveContainer" containerID="d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6" Oct 31 01:29:58.900912 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:29:58.900894 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6\": container with ID starting with d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6 not found: ID does not exist" containerID="d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6" Oct 31 01:29:58.901019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.900920 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6"} err="failed to get container status \"d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6\": rpc error: code = NotFound desc = could not find container \"d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6\": container with ID starting with d42d3708ffa568066e331a078c844be382aaa3741762125b77fd4b261f2435a6 not found: ID does not exist" Oct 31 01:29:58.901019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.900938 5719 scope.go:122] "RemoveContainer" containerID="3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd" Oct 31 01:29:58.909171 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.909151 5719 scope.go:122] "RemoveContainer" containerID="3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd" Oct 31 01:29:58.909410 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:29:58.909392 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd\": container with ID starting with 3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd not found: ID does not exist" containerID="3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd" Oct 31 01:29:58.909486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:58.909416 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd"} err="failed to get container status \"3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd\": rpc error: code = NotFound desc = could not find container \"3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd\": container with ID starting with 3d60f60dc324a3ee7885e833576bd220555f84af5d178bc38e14bc5f4952e5bd not found: ID does not exist" Oct 31 01:29:59.045742 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:59.045715 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj"] Oct 31 01:29:59.051871 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:59.051847 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/simulator-w2-29ece5-kserve-7d6dc9669b-zdkxj"] Oct 31 01:29:59.089084 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:59.089057 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 31 01:29:59.156933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:59.156871 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26"] Oct 31 01:29:59.166238 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:29:59.166216 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-29ece5/premium-w2-29ece5-kserve-999b8b865-r2q26"] Oct 31 01:30:00.199503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.199473 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf"] Oct 31 01:30:00.200169 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200149 5719 cpu_manager.go:414] "containerMap: removing container" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" Oct 31 01:30:00.200257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200170 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" Oct 31 01:30:00.200257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200223 5719 cpu_manager.go:414] "containerMap: removing container" podUID="1c3e65eb-259c-44d6-8711-be9fc63744f0" containerName="revoke-keys" Oct 31 01:30:00.200257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200232 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3e65eb-259c-44d6-8711-be9fc63744f0" containerName="revoke-keys" Oct 31 01:30:00.200257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200244 5719 cpu_manager.go:414] "containerMap: removing container" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" Oct 31 01:30:00.200257 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200252 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" Oct 31 01:30:00.200523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200284 5719 cpu_manager.go:414] "containerMap: removing container" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" Oct 31 01:30:00.200523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200293 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" Oct 31 01:30:00.200523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200395 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" containerName="main" Oct 31 01:30:00.200523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200416 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3e65eb-259c-44d6-8711-be9fc63744f0" containerName="revoke-keys" Oct 31 01:30:00.200523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200427 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a01aef-4f87-4f99-8a1e-b022e707afdb" containerName="main" Oct 31 01:30:00.200523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.200440 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" containerName="main" Oct 31 01:30:00.205256 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.205235 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7"] Oct 31 01:30:00.205383 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.205351 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:00.208719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.208683 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq"] Oct 31 01:30:00.208828 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.208786 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:00.212215 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.212195 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:00.246937 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.246914 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv"] Oct 31 01:30:00.250555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.250528 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf"] Oct 31 01:30:00.250648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.250633 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:00.253875 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.253858 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr"] Oct 31 01:30:00.253988 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.253969 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:00.257153 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.257138 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:00.266158 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.266142 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:30:00.266279 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.266262 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:30:00.292936 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.292895 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:30:00.295653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.295629 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf"] Oct 31 01:30:00.300735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.298374 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7"] Oct 31 01:30:00.300735 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.300261 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq"] Oct 31 01:30:00.350342 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.350292 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf"] Oct 31 01:30:00.355831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.355804 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr"] Oct 31 01:30:00.362234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.362207 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv"] Oct 31 01:30:00.371994 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.371968 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpd7f\" (UniqueName: \"kubernetes.io/projected/c911889e-a973-4356-9ab7-d12b82434283-kube-api-access-lpd7f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf\" (UID: \"c911889e-a973-4356-9ab7-d12b82434283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:00.372131 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372002 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62974\" (UniqueName: \"kubernetes.io/projected/e1750e79-f6b6-47f7-873b-03c0533fcfc9-kube-api-access-62974\") pod \"maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr\" (UID: \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:00.372131 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372023 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjbp\" (UniqueName: \"kubernetes.io/projected/5b106522-b041-4f35-a565-b479b712000a-kube-api-access-lsjbp\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf\" (UID: \"5b106522-b041-4f35-a565-b479b712000a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:00.372131 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372042 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bb8b6e-8992-4fa2-9542-416da1e21e39-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-c95sv\" (UID: \"68bb8b6e-8992-4fa2-9542-416da1e21e39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:00.372268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372184 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq\" (UID: \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:00.372268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372222 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7\" (UID: \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:00.372268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372240 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1750e79-f6b6-47f7-873b-03c0533fcfc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr\" (UID: \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:00.372387 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372312 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thl9b\" (UniqueName: \"kubernetes.io/projected/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-kube-api-access-thl9b\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7\" (UID: \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:00.372387 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372335 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911889e-a973-4356-9ab7-d12b82434283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf\" (UID: \"c911889e-a973-4356-9ab7-d12b82434283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:00.372387 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372369 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4vr\" (UniqueName: \"kubernetes.io/projected/68bb8b6e-8992-4fa2-9542-416da1e21e39-kube-api-access-9r4vr\") pod \"maas-api-key-cleanup-29890170-c95sv\" (UID: \"68bb8b6e-8992-4fa2-9542-416da1e21e39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:00.372545 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372402 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b106522-b041-4f35-a565-b479b712000a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf\" (UID: \"5b106522-b041-4f35-a565-b479b712000a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:00.372545 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.372469 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqgg\" (UniqueName: \"kubernetes.io/projected/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-kube-api-access-hrqgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq\" (UID: \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:00.473745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.473652 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911889e-a973-4356-9ab7-d12b82434283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf\" (UID: \"c911889e-a973-4356-9ab7-d12b82434283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:00.473745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.473731 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4vr\" (UniqueName: \"kubernetes.io/projected/68bb8b6e-8992-4fa2-9542-416da1e21e39-kube-api-access-9r4vr\") pod \"maas-api-key-cleanup-29890170-c95sv\" (UID: \"68bb8b6e-8992-4fa2-9542-416da1e21e39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:00.473974 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.473769 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b106522-b041-4f35-a565-b479b712000a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf\" (UID: \"5b106522-b041-4f35-a565-b479b712000a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:00.473974 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.473790 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqgg\" (UniqueName: \"kubernetes.io/projected/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-kube-api-access-hrqgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq\" (UID: \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:00.473974 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.473876 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpd7f\" (UniqueName: \"kubernetes.io/projected/c911889e-a973-4356-9ab7-d12b82434283-kube-api-access-lpd7f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf\" (UID: \"c911889e-a973-4356-9ab7-d12b82434283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:00.474389 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474363 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62974\" (UniqueName: \"kubernetes.io/projected/e1750e79-f6b6-47f7-873b-03c0533fcfc9-kube-api-access-62974\") pod \"maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr\" (UID: \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:00.474490 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474407 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjbp\" (UniqueName: \"kubernetes.io/projected/5b106522-b041-4f35-a565-b479b712000a-kube-api-access-lsjbp\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf\" (UID: \"5b106522-b041-4f35-a565-b479b712000a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:00.474490 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474413 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911889e-a973-4356-9ab7-d12b82434283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf\" (UID: \"c911889e-a973-4356-9ab7-d12b82434283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:00.474490 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474445 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bb8b6e-8992-4fa2-9542-416da1e21e39-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-c95sv\" (UID: \"68bb8b6e-8992-4fa2-9542-416da1e21e39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:00.474490 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474471 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b106522-b041-4f35-a565-b479b712000a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf\" (UID: \"5b106522-b041-4f35-a565-b479b712000a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:00.474761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474508 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq\" (UID: \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:00.474761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474569 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7\" (UID: \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:00.474761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474599 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1750e79-f6b6-47f7-873b-03c0533fcfc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr\" (UID: \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:00.474761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.474722 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thl9b\" (UniqueName: \"kubernetes.io/projected/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-kube-api-access-thl9b\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7\" (UID: \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:00.475125 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.475104 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bb8b6e-8992-4fa2-9542-416da1e21e39-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-c95sv\" (UID: \"68bb8b6e-8992-4fa2-9542-416da1e21e39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:00.475183 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.475168 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq\" (UID: \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:00.475339 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.475319 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1750e79-f6b6-47f7-873b-03c0533fcfc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr\" (UID: \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:00.475379 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.475319 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7\" (UID: \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:00.534413 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.534390 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4vr\" (UniqueName: \"kubernetes.io/projected/68bb8b6e-8992-4fa2-9542-416da1e21e39-kube-api-access-9r4vr\") pod \"maas-api-key-cleanup-29890170-c95sv\" (UID: \"68bb8b6e-8992-4fa2-9542-416da1e21e39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:00.543281 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.543254 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpd7f\" (UniqueName: \"kubernetes.io/projected/c911889e-a973-4356-9ab7-d12b82434283-kube-api-access-lpd7f\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf\" (UID: \"c911889e-a973-4356-9ab7-d12b82434283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:00.543482 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.543461 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqgg\" (UniqueName: \"kubernetes.io/projected/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-kube-api-access-hrqgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq\" (UID: \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:00.543560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.543542 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjbp\" (UniqueName: \"kubernetes.io/projected/5b106522-b041-4f35-a565-b479b712000a-kube-api-access-lsjbp\") pod \"maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf\" (UID: \"5b106522-b041-4f35-a565-b479b712000a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:00.548518 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.548492 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62974\" (UniqueName: \"kubernetes.io/projected/e1750e79-f6b6-47f7-873b-03c0533fcfc9-kube-api-access-62974\") pod \"maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr\" (UID: \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:00.565146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.565128 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:00.571385 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.571364 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:00.573214 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.573193 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thl9b\" (UniqueName: \"kubernetes.io/projected/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-kube-api-access-thl9b\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7\" (UID: \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:00.578350 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.578329 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:00.604948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.604920 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c759e29-fdad-44b7-a89f-755060c2c483" path="/var/lib/kubelet/pods/3c759e29-fdad-44b7-a89f-755060c2c483/volumes" Oct 31 01:30:00.605416 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.605394 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa568cd6-7a3e-4f31-8271-781a4f9d4318" path="/var/lib/kubelet/pods/fa568cd6-7a3e-4f31-8271-781a4f9d4318/volumes" Oct 31 01:30:00.820144 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.820119 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:00.827152 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.827132 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:00.834056 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.834037 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:00.959118 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:00.959089 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf"] Oct 31 01:30:00.964775 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:00.964745 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b106522_b041_4f35_a565_b479b712000a.slice/crio-de5d46a6015c87e68c46a90abb9307d9c397cc3368233b7202aa143eaceffc2d WatchSource:0}: Error finding container de5d46a6015c87e68c46a90abb9307d9c397cc3368233b7202aa143eaceffc2d: Status 404 returned error can't find the container with id de5d46a6015c87e68c46a90abb9307d9c397cc3368233b7202aa143eaceffc2d Oct 31 01:30:01.009128 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.009106 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr"] Oct 31 01:30:01.014991 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:01.014964 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1750e79_f6b6_47f7_873b_03c0533fcfc9.slice/crio-c7533666245590a921d149d43147f29e726e639ef10944f8f4b2f4d47ed5de12 WatchSource:0}: Error finding container c7533666245590a921d149d43147f29e726e639ef10944f8f4b2f4d47ed5de12: Status 404 returned error can't find the container with id c7533666245590a921d149d43147f29e726e639ef10944f8f4b2f4d47ed5de12 Oct 31 01:30:01.081974 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.081911 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv"] Oct 31 01:30:01.087397 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:01.087374 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bb8b6e_8992_4fa2_9542_416da1e21e39.slice/crio-8c50cb40b394dfc80b666116fda46e3e1f02aa118e382fe9ea61bf52356d62f3 WatchSource:0}: Error finding container 8c50cb40b394dfc80b666116fda46e3e1f02aa118e382fe9ea61bf52356d62f3: Status 404 returned error can't find the container with id 8c50cb40b394dfc80b666116fda46e3e1f02aa118e382fe9ea61bf52356d62f3 Oct 31 01:30:01.195387 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.195360 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf"] Oct 31 01:30:01.202724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.202682 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7"] Oct 31 01:30:01.203122 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:01.202988 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc911889e_a973_4356_9ab7_d12b82434283.slice/crio-85f547c22c670438b2926579de5108dde55c1614a2040f5005b6c71c31142476 WatchSource:0}: Error finding container 85f547c22c670438b2926579de5108dde55c1614a2040f5005b6c71c31142476: Status 404 returned error can't find the container with id 85f547c22c670438b2926579de5108dde55c1614a2040f5005b6c71c31142476 Oct 31 01:30:01.214618 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:01.214599 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7079fd16_ce2b_466b_afb3_fa4a7dd8a7ff.slice/crio-babba0eecd9b6749a3e5d6b911180e6a62c0f2bf1181dfbcadcb6c0d516c038f WatchSource:0}: Error finding container babba0eecd9b6749a3e5d6b911180e6a62c0f2bf1181dfbcadcb6c0d516c038f: Status 404 returned error can't find the container with id babba0eecd9b6749a3e5d6b911180e6a62c0f2bf1181dfbcadcb6c0d516c038f Oct 31 01:30:01.226788 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.226765 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq"] Oct 31 01:30:01.233841 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:01.233815 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e651e26_89f2_4bab_bd11_8291fc0ac2c0.slice/crio-f816b642004bf744e286d1f2e8da70dd0d81753f3a785d97d1baa90a3ea90a27 WatchSource:0}: Error finding container f816b642004bf744e286d1f2e8da70dd0d81753f3a785d97d1baa90a3ea90a27: Status 404 returned error can't find the container with id f816b642004bf744e286d1f2e8da70dd0d81753f3a785d97d1baa90a3ea90a27 Oct 31 01:30:01.764112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.764045 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" podUID="176128e0-a162-4957-8fc3-aba7a4076be7" containerName="payload-processing" containerID="cri-o://02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232" gracePeriod=25 Oct 31 01:30:01.873012 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.872977 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" podUID="19a35525-1b77-4772-b089-b8a786ab452b" containerName="payload-pre-processing" containerID="cri-o://b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29" gracePeriod=25 Oct 31 01:30:01.922362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.922329 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" event={"ID":"c911889e-a973-4356-9ab7-d12b82434283","Type":"ContainerStarted","Data":"8fb0cf639e1c980af4163fe4d7d5adb4678ef88ea12377a95453b708904ffb6d"} Oct 31 01:30:01.922486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.922419 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" event={"ID":"c911889e-a973-4356-9ab7-d12b82434283","Type":"ContainerStarted","Data":"85f547c22c670438b2926579de5108dde55c1614a2040f5005b6c71c31142476"} Oct 31 01:30:01.924102 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.924068 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" event={"ID":"5e651e26-89f2-4bab-bd11-8291fc0ac2c0","Type":"ContainerStarted","Data":"ea6fec4b5260f46e9fb4c15e8314ce55d715fab051ee459afb0c26a3be5ad7c7"} Oct 31 01:30:01.924213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.924122 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" event={"ID":"5e651e26-89f2-4bab-bd11-8291fc0ac2c0","Type":"ContainerStarted","Data":"f816b642004bf744e286d1f2e8da70dd0d81753f3a785d97d1baa90a3ea90a27"} Oct 31 01:30:01.925736 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.925713 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" event={"ID":"5b106522-b041-4f35-a565-b479b712000a","Type":"ContainerStarted","Data":"72fe0dcc795906ffd8120cde2c048738bc3995aa10830daff90be34a00cca0cd"} Oct 31 01:30:01.925841 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.925751 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" event={"ID":"5b106522-b041-4f35-a565-b479b712000a","Type":"ContainerStarted","Data":"de5d46a6015c87e68c46a90abb9307d9c397cc3368233b7202aa143eaceffc2d"} Oct 31 01:30:01.925888 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.925854 5719 scope.go:122] "RemoveContainer" containerID="72fe0dcc795906ffd8120cde2c048738bc3995aa10830daff90be34a00cca0cd" Oct 31 01:30:01.927435 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.927409 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" event={"ID":"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff","Type":"ContainerStarted","Data":"256b0dcabcbe24ddfbb8a7bfeee5b90740dcdeaa1e4ab8c2052b39ee064fd0ca"} Oct 31 01:30:01.927563 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.927456 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" event={"ID":"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff","Type":"ContainerStarted","Data":"babba0eecd9b6749a3e5d6b911180e6a62c0f2bf1181dfbcadcb6c0d516c038f"} Oct 31 01:30:01.929299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.929262 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" event={"ID":"e1750e79-f6b6-47f7-873b-03c0533fcfc9","Type":"ContainerStarted","Data":"ba7d00aa9569a83c23976133eded25be692fcfd494acf452dde940e3a9a3bedc"} Oct 31 01:30:01.929446 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.929426 5719 scope.go:122] "RemoveContainer" containerID="ba7d00aa9569a83c23976133eded25be692fcfd494acf452dde940e3a9a3bedc" Oct 31 01:30:01.929520 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.929438 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" event={"ID":"e1750e79-f6b6-47f7-873b-03c0533fcfc9","Type":"ContainerStarted","Data":"c7533666245590a921d149d43147f29e726e639ef10944f8f4b2f4d47ed5de12"} Oct 31 01:30:01.931002 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.930979 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" event={"ID":"68bb8b6e-8992-4fa2-9542-416da1e21e39","Type":"ContainerStarted","Data":"520a02076cc5dfb69ee13e5e0097e396d723d55f524f827760cc94391c1b8e77"} Oct 31 01:30:01.931076 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:01.931025 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" event={"ID":"68bb8b6e-8992-4fa2-9542-416da1e21e39","Type":"ContainerStarted","Data":"8c50cb40b394dfc80b666116fda46e3e1f02aa118e382fe9ea61bf52356d62f3"} Oct 31 01:30:02.034643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.034600 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" podStartSLOduration=2.034587975 podStartE2EDuration="2.034587975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.034089947 +0000 UTC m=+3683.570232252" watchObservedRunningTime="2026-10-31 01:30:02.034587975 +0000 UTC m=+3683.570730280" Oct 31 01:30:02.148948 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.148904 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" podStartSLOduration=2.148891197 podStartE2EDuration="2.148891197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.147277762 +0000 UTC m=+3683.683420067" watchObservedRunningTime="2026-10-31 01:30:02.148891197 +0000 UTC m=+3683.685033504" Oct 31 01:30:02.372168 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.372121 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" podStartSLOduration=2.372108436 podStartE2EDuration="2.372108436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.370900431 +0000 UTC m=+3683.907042735" watchObservedRunningTime="2026-10-31 01:30:02.372108436 +0000 UTC m=+3683.908250741" Oct 31 01:30:02.592505 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.592460 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" podStartSLOduration=2.592442298 podStartE2EDuration="2.592442298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.591364455 +0000 UTC m=+3684.127506761" watchObservedRunningTime="2026-10-31 01:30:02.592442298 +0000 UTC m=+3684.128584604" Oct 31 01:30:02.939023 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.938988 5719 generic.go:356] "Generic (PLEG): container finished" podID="c911889e-a973-4356-9ab7-d12b82434283" containerID="8fb0cf639e1c980af4163fe4d7d5adb4678ef88ea12377a95453b708904ffb6d" exitCode=60 Oct 31 01:30:02.939198 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.939077 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" event={"ID":"c911889e-a973-4356-9ab7-d12b82434283","Type":"ContainerDied","Data":"8fb0cf639e1c980af4163fe4d7d5adb4678ef88ea12377a95453b708904ffb6d"} Oct 31 01:30:02.939331 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.939312 5719 scope.go:122] "RemoveContainer" containerID="8fb0cf639e1c980af4163fe4d7d5adb4678ef88ea12377a95453b708904ffb6d" Oct 31 01:30:02.941004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.940836 5719 generic.go:356] "Generic (PLEG): container finished" podID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerID="ea6fec4b5260f46e9fb4c15e8314ce55d715fab051ee459afb0c26a3be5ad7c7" exitCode=60 Oct 31 01:30:02.941004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.940914 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" event={"ID":"5e651e26-89f2-4bab-bd11-8291fc0ac2c0","Type":"ContainerDied","Data":"ea6fec4b5260f46e9fb4c15e8314ce55d715fab051ee459afb0c26a3be5ad7c7"} Oct 31 01:30:02.941143 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.941127 5719 scope.go:122] "RemoveContainer" containerID="ea6fec4b5260f46e9fb4c15e8314ce55d715fab051ee459afb0c26a3be5ad7c7" Oct 31 01:30:02.943439 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.943416 5719 generic.go:356] "Generic (PLEG): container finished" podID="5b106522-b041-4f35-a565-b479b712000a" containerID="72fe0dcc795906ffd8120cde2c048738bc3995aa10830daff90be34a00cca0cd" exitCode=60 Oct 31 01:30:02.943556 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.943532 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" event={"ID":"5b106522-b041-4f35-a565-b479b712000a","Type":"ContainerDied","Data":"72fe0dcc795906ffd8120cde2c048738bc3995aa10830daff90be34a00cca0cd"} Oct 31 01:30:02.943618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.943579 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" event={"ID":"5b106522-b041-4f35-a565-b479b712000a","Type":"ContainerStarted","Data":"fdf76a925342bc5b002bef0df1846944e964a3c533e28dd925ab82d5ffe4f9bc"} Oct 31 01:30:02.945264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.945240 5719 generic.go:356] "Generic (PLEG): container finished" podID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerID="256b0dcabcbe24ddfbb8a7bfeee5b90740dcdeaa1e4ab8c2052b39ee064fd0ca" exitCode=60 Oct 31 01:30:02.945372 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.945270 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" event={"ID":"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff","Type":"ContainerDied","Data":"256b0dcabcbe24ddfbb8a7bfeee5b90740dcdeaa1e4ab8c2052b39ee064fd0ca"} Oct 31 01:30:02.945525 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.945507 5719 scope.go:122] "RemoveContainer" containerID="256b0dcabcbe24ddfbb8a7bfeee5b90740dcdeaa1e4ab8c2052b39ee064fd0ca" Oct 31 01:30:02.947170 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.947147 5719 generic.go:356] "Generic (PLEG): container finished" podID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerID="ba7d00aa9569a83c23976133eded25be692fcfd494acf452dde940e3a9a3bedc" exitCode=60 Oct 31 01:30:02.947306 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.947203 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" event={"ID":"e1750e79-f6b6-47f7-873b-03c0533fcfc9","Type":"ContainerDied","Data":"ba7d00aa9569a83c23976133eded25be692fcfd494acf452dde940e3a9a3bedc"} Oct 31 01:30:02.947306 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.947262 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" event={"ID":"e1750e79-f6b6-47f7-873b-03c0533fcfc9","Type":"ContainerStarted","Data":"55b1377dd53ca8d93977ac11ac5e4634d3c3a3d789947656ebfee9dfc6c5ce90"} Oct 31 01:30:02.948939 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.948917 5719 generic.go:356] "Generic (PLEG): container finished" podID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerID="520a02076cc5dfb69ee13e5e0097e396d723d55f524f827760cc94391c1b8e77" exitCode=60 Oct 31 01:30:02.949026 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.948976 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" event={"ID":"68bb8b6e-8992-4fa2-9542-416da1e21e39","Type":"ContainerDied","Data":"520a02076cc5dfb69ee13e5e0097e396d723d55f524f827760cc94391c1b8e77"} Oct 31 01:30:02.949213 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:02.949196 5719 scope.go:122] "RemoveContainer" containerID="520a02076cc5dfb69ee13e5e0097e396d723d55f524f827760cc94391c1b8e77" Oct 31 01:30:03.010809 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:03.010767 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:30:03.288523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.288477 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" podStartSLOduration=3.288462157 podStartE2EDuration="3.288462157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:03.287215587 +0000 UTC m=+3684.823357891" watchObservedRunningTime="2026-10-31 01:30:03.288462157 +0000 UTC m=+3684.824604462" Oct 31 01:30:03.511643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.511571 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" podStartSLOduration=3.511553578 podStartE2EDuration="3.511553578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:03.510386063 +0000 UTC m=+3685.046528369" watchObservedRunningTime="2026-10-31 01:30:03.511553578 +0000 UTC m=+3685.047695944" Oct 31 01:30:03.960406 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.960380 5719 generic.go:356] "Generic (PLEG): container finished" podID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerID="7da210fedc6236a93f6e497a6256e8fef902202f236491620a563818b880a9ea" exitCode=60 Oct 31 01:30:03.960524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.960429 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" event={"ID":"68bb8b6e-8992-4fa2-9542-416da1e21e39","Type":"ContainerDied","Data":"7da210fedc6236a93f6e497a6256e8fef902202f236491620a563818b880a9ea"} Oct 31 01:30:03.960524 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.960488 5719 scope.go:122] "RemoveContainer" containerID="520a02076cc5dfb69ee13e5e0097e396d723d55f524f827760cc94391c1b8e77" Oct 31 01:30:03.960734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.960716 5719 scope.go:122] "RemoveContainer" containerID="7da210fedc6236a93f6e497a6256e8fef902202f236491620a563818b880a9ea" Oct 31 01:30:03.961021 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:03.960994 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-c95sv_odh-ai-gateway-infra(68bb8b6e-8992-4fa2-9542-416da1e21e39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" Oct 31 01:30:03.962653 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.962631 5719 generic.go:356] "Generic (PLEG): container finished" podID="c911889e-a973-4356-9ab7-d12b82434283" containerID="545022bb32939c2de3d13b079ed9619631104b05c0060f856701024740d86654" exitCode=60 Oct 31 01:30:03.962754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.962720 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" event={"ID":"c911889e-a973-4356-9ab7-d12b82434283","Type":"ContainerDied","Data":"545022bb32939c2de3d13b079ed9619631104b05c0060f856701024740d86654"} Oct 31 01:30:03.962957 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.962944 5719 scope.go:122] "RemoveContainer" containerID="545022bb32939c2de3d13b079ed9619631104b05c0060f856701024740d86654" Oct 31 01:30:03.963179 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:03.963160 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf_odh-ai-gateway-infra(c911889e-a973-4356-9ab7-d12b82434283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" podUID="c911889e-a973-4356-9ab7-d12b82434283" Oct 31 01:30:03.964749 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.964728 5719 generic.go:356] "Generic (PLEG): container finished" podID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerID="c8b4e406cdbe86410e7a1d0ab3b9db1788aeb251870c95d45853fd531c9f394b" exitCode=60 Oct 31 01:30:03.964848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.964815 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" event={"ID":"5e651e26-89f2-4bab-bd11-8291fc0ac2c0","Type":"ContainerDied","Data":"c8b4e406cdbe86410e7a1d0ab3b9db1788aeb251870c95d45853fd531c9f394b"} Oct 31 01:30:03.965020 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.965004 5719 scope.go:122] "RemoveContainer" containerID="c8b4e406cdbe86410e7a1d0ab3b9db1788aeb251870c95d45853fd531c9f394b" Oct 31 01:30:03.965235 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:03.965214 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq_odh-ai-gateway-infra(5e651e26-89f2-4bab-bd11-8291fc0ac2c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" Oct 31 01:30:03.966976 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.966954 5719 generic.go:356] "Generic (PLEG): container finished" podID="5b106522-b041-4f35-a565-b479b712000a" containerID="fdf76a925342bc5b002bef0df1846944e964a3c533e28dd925ab82d5ffe4f9bc" exitCode=60 Oct 31 01:30:03.967104 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.967082 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" event={"ID":"5b106522-b041-4f35-a565-b479b712000a","Type":"ContainerDied","Data":"fdf76a925342bc5b002bef0df1846944e964a3c533e28dd925ab82d5ffe4f9bc"} Oct 31 01:30:03.967236 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.967220 5719 scope.go:122] "RemoveContainer" containerID="fdf76a925342bc5b002bef0df1846944e964a3c533e28dd925ab82d5ffe4f9bc" Oct 31 01:30:03.967417 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:03.967395 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf_odh-ai-gateway-infra(5b106522-b041-4f35-a565-b479b712000a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" podUID="5b106522-b041-4f35-a565-b479b712000a" Oct 31 01:30:03.969040 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.969020 5719 generic.go:356] "Generic (PLEG): container finished" podID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerID="81d78cb6c50a5062c263a1b52f2820d364a735692a2fe262b83ffc27c733e3cb" exitCode=60 Oct 31 01:30:03.969140 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.969088 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" event={"ID":"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff","Type":"ContainerDied","Data":"81d78cb6c50a5062c263a1b52f2820d364a735692a2fe262b83ffc27c733e3cb"} Oct 31 01:30:03.969314 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.969298 5719 scope.go:122] "RemoveContainer" containerID="81d78cb6c50a5062c263a1b52f2820d364a735692a2fe262b83ffc27c733e3cb" Oct 31 01:30:03.969514 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:03.969482 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7_odh-ai-gateway-infra(7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" Oct 31 01:30:03.970971 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.970952 5719 generic.go:356] "Generic (PLEG): container finished" podID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerID="55b1377dd53ca8d93977ac11ac5e4634d3c3a3d789947656ebfee9dfc6c5ce90" exitCode=60 Oct 31 01:30:03.971049 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.970977 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" event={"ID":"e1750e79-f6b6-47f7-873b-03c0533fcfc9","Type":"ContainerDied","Data":"55b1377dd53ca8d93977ac11ac5e4634d3c3a3d789947656ebfee9dfc6c5ce90"} Oct 31 01:30:03.971172 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:03.971159 5719 scope.go:122] "RemoveContainer" containerID="55b1377dd53ca8d93977ac11ac5e4634d3c3a3d789947656ebfee9dfc6c5ce90" Oct 31 01:30:03.971320 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:03.971303 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr_odh-ai-gateway-infra(e1750e79-f6b6-47f7-873b-03c0533fcfc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" Oct 31 01:30:04.021397 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.021379 5719 scope.go:122] "RemoveContainer" containerID="8fb0cf639e1c980af4163fe4d7d5adb4678ef88ea12377a95453b708904ffb6d" Oct 31 01:30:04.033682 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.033657 5719 scope.go:122] "RemoveContainer" containerID="ea6fec4b5260f46e9fb4c15e8314ce55d715fab051ee459afb0c26a3be5ad7c7" Oct 31 01:30:04.045486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.045456 5719 scope.go:122] "RemoveContainer" containerID="72fe0dcc795906ffd8120cde2c048738bc3995aa10830daff90be34a00cca0cd" Oct 31 01:30:04.057303 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.057283 5719 scope.go:122] "RemoveContainer" containerID="256b0dcabcbe24ddfbb8a7bfeee5b90740dcdeaa1e4ab8c2052b39ee064fd0ca" Oct 31 01:30:04.069586 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.069568 5719 scope.go:122] "RemoveContainer" containerID="ba7d00aa9569a83c23976133eded25be692fcfd494acf452dde940e3a9a3bedc" Oct 31 01:30:04.304745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.304684 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" podUID="176128e0-a162-4957-8fc3-aba7a4076be7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:04.979926 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.979899 5719 scope.go:122] "RemoveContainer" containerID="c8b4e406cdbe86410e7a1d0ab3b9db1788aeb251870c95d45853fd531c9f394b" Oct 31 01:30:04.980422 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:04.980164 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq_odh-ai-gateway-infra(5e651e26-89f2-4bab-bd11-8291fc0ac2c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" Oct 31 01:30:04.981594 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.981574 5719 scope.go:122] "RemoveContainer" containerID="fdf76a925342bc5b002bef0df1846944e964a3c533e28dd925ab82d5ffe4f9bc" Oct 31 01:30:04.981807 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:04.981788 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf_odh-ai-gateway-infra(5b106522-b041-4f35-a565-b479b712000a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" podUID="5b106522-b041-4f35-a565-b479b712000a" Oct 31 01:30:04.983116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.983095 5719 scope.go:122] "RemoveContainer" containerID="81d78cb6c50a5062c263a1b52f2820d364a735692a2fe262b83ffc27c733e3cb" Oct 31 01:30:04.983286 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:04.983267 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7_odh-ai-gateway-infra(7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" Oct 31 01:30:04.984606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.984590 5719 scope.go:122] "RemoveContainer" containerID="55b1377dd53ca8d93977ac11ac5e4634d3c3a3d789947656ebfee9dfc6c5ce90" Oct 31 01:30:04.984831 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:04.984808 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr_odh-ai-gateway-infra(e1750e79-f6b6-47f7-873b-03c0533fcfc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" Oct 31 01:30:04.986109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.986095 5719 scope.go:122] "RemoveContainer" containerID="7da210fedc6236a93f6e497a6256e8fef902202f236491620a563818b880a9ea" Oct 31 01:30:04.986296 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:04.986274 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-c95sv_odh-ai-gateway-infra(68bb8b6e-8992-4fa2-9542-416da1e21e39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" Oct 31 01:30:04.987599 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:04.987585 5719 scope.go:122] "RemoveContainer" containerID="545022bb32939c2de3d13b079ed9619631104b05c0060f856701024740d86654" Oct 31 01:30:04.987769 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:04.987751 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf_odh-ai-gateway-infra(c911889e-a973-4356-9ab7-d12b82434283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" podUID="c911889e-a973-4356-9ab7-d12b82434283" Oct 31 01:30:05.310574 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.310494 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" podUID="19a35525-1b77-4772-b089-b8a786ab452b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:05.446082 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.446052 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx"] Oct 31 01:30:05.449755 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.449739 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.506244 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.506220 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx"] Oct 31 01:30:05.550801 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.550770 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:05.626093 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.626053 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.626314 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.626113 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kwgr\" (UniqueName: \"kubernetes.io/projected/269c38ce-d974-4242-aa15-21af8c116ddc-kube-api-access-2kwgr\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.626314 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.626135 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.727044 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.727005 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.727186 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.727167 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kwgr\" (UniqueName: \"kubernetes.io/projected/269c38ce-d974-4242-aa15-21af8c116ddc-kube-api-access-2kwgr\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.727244 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.727216 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.727677 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.727659 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.729632 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.729608 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:05.786089 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:05.786063 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kwgr\" (UniqueName: \"kubernetes.io/projected/269c38ce-d974-4242-aa15-21af8c116ddc-kube-api-access-2kwgr\") pod \"maas-api-revoke-sub-b678854d-w47qx\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:06.064164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:06.064100 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:06.436890 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:06.436855 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx"] Oct 31 01:30:06.444492 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:06.444456 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269c38ce_d974_4242_aa15_21af8c116ddc.slice/crio-65f5042b7c2af674cd7265b9ba894535e201d62d984780e3c8a548d9189e41ff WatchSource:0}: Error finding container 65f5042b7c2af674cd7265b9ba894535e201d62d984780e3c8a548d9189e41ff: Status 404 returned error can't find the container with id 65f5042b7c2af674cd7265b9ba894535e201d62d984780e3c8a548d9189e41ff Oct 31 01:30:07.001122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:07.001024 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" event={"ID":"269c38ce-d974-4242-aa15-21af8c116ddc","Type":"ContainerStarted","Data":"2d2125fc630e96b2a5e8fb593842dc6ca3ed6e0e43967179f38633a9518793ff"} Oct 31 01:30:07.001122 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:07.001080 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" event={"ID":"269c38ce-d974-4242-aa15-21af8c116ddc","Type":"ContainerStarted","Data":"65f5042b7c2af674cd7265b9ba894535e201d62d984780e3c8a548d9189e41ff"} Oct 31 01:30:07.113870 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:07.113813 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" podStartSLOduration=2.113798883 podStartE2EDuration="2.113798883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:07.11318632 +0000 UTC m=+3688.649328628" watchObservedRunningTime="2026-10-31 01:30:07.113798883 +0000 UTC m=+3688.649941188" Oct 31 01:30:08.008963 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:08.008922 5719 generic.go:356] "Generic (PLEG): container finished" podID="269c38ce-d974-4242-aa15-21af8c116ddc" containerID="2d2125fc630e96b2a5e8fb593842dc6ca3ed6e0e43967179f38633a9518793ff" exitCode=0 Oct 31 01:30:08.009142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:08.009012 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" event={"ID":"269c38ce-d974-4242-aa15-21af8c116ddc","Type":"ContainerDied","Data":"2d2125fc630e96b2a5e8fb593842dc6ca3ed6e0e43967179f38633a9518793ff"} Oct 31 01:30:09.162744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.162720 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:09.359008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.358984 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269c38ce-d974-4242-aa15-21af8c116ddc-kube-api-access-2kwgr\" (UniqueName: \"kubernetes.io/projected/269c38ce-d974-4242-aa15-21af8c116ddc-kube-api-access-2kwgr\") pod \"269c38ce-d974-4242-aa15-21af8c116ddc\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " Oct 31 01:30:09.359211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.359029 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-xks-ca\") pod \"269c38ce-d974-4242-aa15-21af8c116ddc\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " Oct 31 01:30:09.359304 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.359221 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-openshift-ca\") pod \"269c38ce-d974-4242-aa15-21af8c116ddc\" (UID: \"269c38ce-d974-4242-aa15-21af8c116ddc\") " Oct 31 01:30:09.359660 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.359629 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-openshift-ca" pod "269c38ce-d974-4242-aa15-21af8c116ddc" (UID: "269c38ce-d974-4242-aa15-21af8c116ddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:09.359867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.359849 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:09.361448 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.361420 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269c38ce-d974-4242-aa15-21af8c116ddc-kube-api-access-2kwgr" pod "269c38ce-d974-4242-aa15-21af8c116ddc" (UID: "269c38ce-d974-4242-aa15-21af8c116ddc"). InnerVolumeSpecName "kube-api-access-2kwgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:09.361906 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.361883 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-xks-ca" pod "269c38ce-d974-4242-aa15-21af8c116ddc" (UID: "269c38ce-d974-4242-aa15-21af8c116ddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:09.461190 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.461164 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kwgr\" (UniqueName: \"kubernetes.io/projected/269c38ce-d974-4242-aa15-21af8c116ddc-kube-api-access-2kwgr\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:09.461278 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:09.461194 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/269c38ce-d974-4242-aa15-21af8c116ddc-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:10.023982 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:10.023954 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" Oct 31 01:30:10.024139 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:10.024001 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx" event={"ID":"269c38ce-d974-4242-aa15-21af8c116ddc","Type":"ContainerDied","Data":"65f5042b7c2af674cd7265b9ba894535e201d62d984780e3c8a548d9189e41ff"} Oct 31 01:30:10.024139 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:10.024053 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65f5042b7c2af674cd7265b9ba894535e201d62d984780e3c8a548d9189e41ff" Oct 31 01:30:13.595501 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:13.595473 5719 scope.go:122] "RemoveContainer" containerID="55b1377dd53ca8d93977ac11ac5e4634d3c3a3d789947656ebfee9dfc6c5ce90" Oct 31 01:30:14.049916 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:14.049888 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" event={"ID":"e1750e79-f6b6-47f7-873b-03c0533fcfc9","Type":"ContainerStarted","Data":"ecee6e38959f7140dea205c66587dd11e751f658a5a018e5a7fc0a7f4cfde2aa"} Oct 31 01:30:14.303719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:14.303620 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" podUID="176128e0-a162-4957-8fc3-aba7a4076be7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:14.595580 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:14.595554 5719 scope.go:122] "RemoveContainer" containerID="fdf76a925342bc5b002bef0df1846944e964a3c533e28dd925ab82d5ffe4f9bc" Oct 31 01:30:15.057595 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:15.057557 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" event={"ID":"5b106522-b041-4f35-a565-b479b712000a","Type":"ContainerStarted","Data":"22278c40cd18f31fdac083ed0ed63c81a63169cf88fbb853880f1201ccfe8c21"} Oct 31 01:30:15.059522 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:15.059495 5719 generic.go:356] "Generic (PLEG): container finished" podID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerID="ecee6e38959f7140dea205c66587dd11e751f658a5a018e5a7fc0a7f4cfde2aa" exitCode=60 Oct 31 01:30:15.059661 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:15.059564 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" event={"ID":"e1750e79-f6b6-47f7-873b-03c0533fcfc9","Type":"ContainerDied","Data":"ecee6e38959f7140dea205c66587dd11e751f658a5a018e5a7fc0a7f4cfde2aa"} Oct 31 01:30:15.059661 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:15.059617 5719 scope.go:122] "RemoveContainer" containerID="55b1377dd53ca8d93977ac11ac5e4634d3c3a3d789947656ebfee9dfc6c5ce90" Oct 31 01:30:15.059927 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:15.059912 5719 scope.go:122] "RemoveContainer" containerID="ecee6e38959f7140dea205c66587dd11e751f658a5a018e5a7fc0a7f4cfde2aa" Oct 31 01:30:15.060181 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:15.060152 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr_odh-ai-gateway-infra(e1750e79-f6b6-47f7-873b-03c0533fcfc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" Oct 31 01:30:15.174414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:15.174348 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr"] Oct 31 01:30:15.310765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:15.310730 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" podUID="19a35525-1b77-4772-b089-b8a786ab452b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:30:15.438861 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:15.438806 5719 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b106522_b041_4f35_a565_b479b712000a.slice/crio-22278c40cd18f31fdac083ed0ed63c81a63169cf88fbb853880f1201ccfe8c21.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:30:16.068486 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.068454 5719 generic.go:356] "Generic (PLEG): container finished" podID="5b106522-b041-4f35-a565-b479b712000a" containerID="22278c40cd18f31fdac083ed0ed63c81a63169cf88fbb853880f1201ccfe8c21" exitCode=60 Oct 31 01:30:16.068983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.068487 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" event={"ID":"5b106522-b041-4f35-a565-b479b712000a","Type":"ContainerDied","Data":"22278c40cd18f31fdac083ed0ed63c81a63169cf88fbb853880f1201ccfe8c21"} Oct 31 01:30:16.068983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.068554 5719 scope.go:122] "RemoveContainer" containerID="fdf76a925342bc5b002bef0df1846944e964a3c533e28dd925ab82d5ffe4f9bc" Oct 31 01:30:16.068983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.068955 5719 scope.go:122] "RemoveContainer" containerID="22278c40cd18f31fdac083ed0ed63c81a63169cf88fbb853880f1201ccfe8c21" Oct 31 01:30:16.069314 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:16.069250 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf_odh-ai-gateway-infra(5b106522-b041-4f35-a565-b479b712000a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" podUID="5b106522-b041-4f35-a565-b479b712000a" Oct 31 01:30:16.178862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.178829 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf"] Oct 31 01:30:16.210376 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.210352 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:16.328449 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.328374 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1750e79-f6b6-47f7-873b-03c0533fcfc9-kube-api-access-62974\" (UniqueName: \"kubernetes.io/projected/e1750e79-f6b6-47f7-873b-03c0533fcfc9-kube-api-access-62974\") pod \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\" (UID: \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\") " Oct 31 01:30:16.328604 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.328524 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1750e79-f6b6-47f7-873b-03c0533fcfc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1750e79-f6b6-47f7-873b-03c0533fcfc9-maas-api-service-ca\") pod \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\" (UID: \"e1750e79-f6b6-47f7-873b-03c0533fcfc9\") " Oct 31 01:30:16.328867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.328845 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1750e79-f6b6-47f7-873b-03c0533fcfc9-maas-api-service-ca" pod "e1750e79-f6b6-47f7-873b-03c0533fcfc9" (UID: "e1750e79-f6b6-47f7-873b-03c0533fcfc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:16.330667 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.330644 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1750e79-f6b6-47f7-873b-03c0533fcfc9-kube-api-access-62974" pod "e1750e79-f6b6-47f7-873b-03c0533fcfc9" (UID: "e1750e79-f6b6-47f7-873b-03c0533fcfc9"). InnerVolumeSpecName "kube-api-access-62974". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:16.429536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.429507 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1750e79-f6b6-47f7-873b-03c0533fcfc9-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:16.429536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.429532 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62974\" (UniqueName: \"kubernetes.io/projected/e1750e79-f6b6-47f7-873b-03c0533fcfc9-kube-api-access-62974\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:16.595055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:16.594986 5719 scope.go:122] "RemoveContainer" containerID="c8b4e406cdbe86410e7a1d0ab3b9db1788aeb251870c95d45853fd531c9f394b" Oct 31 01:30:17.078988 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.078956 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" event={"ID":"5e651e26-89f2-4bab-bd11-8291fc0ac2c0","Type":"ContainerStarted","Data":"b3b86c436d4d5c48dc7483af246c57cf027b2202dcf2800558d50dfb1743029f"} Oct 31 01:30:17.082238 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.082219 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" Oct 31 01:30:17.082346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.082218 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr" event={"ID":"e1750e79-f6b6-47f7-873b-03c0533fcfc9","Type":"ContainerDied","Data":"c7533666245590a921d149d43147f29e726e639ef10944f8f4b2f4d47ed5de12"} Oct 31 01:30:17.082414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.082367 5719 scope.go:122] "RemoveContainer" containerID="ecee6e38959f7140dea205c66587dd11e751f658a5a018e5a7fc0a7f4cfde2aa" Oct 31 01:30:17.225382 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.225357 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:17.338509 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.338485 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b106522-b041-4f35-a565-b479b712000a-kube-api-access-lsjbp\" (UniqueName: \"kubernetes.io/projected/5b106522-b041-4f35-a565-b479b712000a-kube-api-access-lsjbp\") pod \"5b106522-b041-4f35-a565-b479b712000a\" (UID: \"5b106522-b041-4f35-a565-b479b712000a\") " Oct 31 01:30:17.338641 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.338572 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b106522-b041-4f35-a565-b479b712000a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b106522-b041-4f35-a565-b479b712000a-maas-api-service-ca\") pod \"5b106522-b041-4f35-a565-b479b712000a\" (UID: \"5b106522-b041-4f35-a565-b479b712000a\") " Oct 31 01:30:17.338935 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.338912 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b106522-b041-4f35-a565-b479b712000a-maas-api-service-ca" pod "5b106522-b041-4f35-a565-b479b712000a" (UID: "5b106522-b041-4f35-a565-b479b712000a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:17.340716 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.340678 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b106522-b041-4f35-a565-b479b712000a-kube-api-access-lsjbp" pod "5b106522-b041-4f35-a565-b479b712000a" (UID: "5b106522-b041-4f35-a565-b479b712000a"). InnerVolumeSpecName "kube-api-access-lsjbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:17.439737 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.439716 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b106522-b041-4f35-a565-b479b712000a-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:17.439832 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.439738 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsjbp\" (UniqueName: \"kubernetes.io/projected/5b106522-b041-4f35-a565-b479b712000a-kube-api-access-lsjbp\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:17.471687 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.471639 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr"] Oct 31 01:30:17.477499 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:17.477479 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c61da9-29890170-qj8vr"] Oct 31 01:30:18.090199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.090169 5719 generic.go:356] "Generic (PLEG): container finished" podID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerID="b3b86c436d4d5c48dc7483af246c57cf027b2202dcf2800558d50dfb1743029f" exitCode=60 Oct 31 01:30:18.090676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.090236 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" event={"ID":"5e651e26-89f2-4bab-bd11-8291fc0ac2c0","Type":"ContainerDied","Data":"b3b86c436d4d5c48dc7483af246c57cf027b2202dcf2800558d50dfb1743029f"} Oct 31 01:30:18.090676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.090293 5719 scope.go:122] "RemoveContainer" containerID="c8b4e406cdbe86410e7a1d0ab3b9db1788aeb251870c95d45853fd531c9f394b" Oct 31 01:30:18.090676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.090631 5719 scope.go:122] "RemoveContainer" containerID="b3b86c436d4d5c48dc7483af246c57cf027b2202dcf2800558d50dfb1743029f" Oct 31 01:30:18.090922 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:18.090881 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq_odh-ai-gateway-infra(5e651e26-89f2-4bab-bd11-8291fc0ac2c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" Oct 31 01:30:18.091879 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.091858 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" Oct 31 01:30:18.091959 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.091876 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf" event={"ID":"5b106522-b041-4f35-a565-b479b712000a","Type":"ContainerDied","Data":"de5d46a6015c87e68c46a90abb9307d9c397cc3368233b7202aa143eaceffc2d"} Oct 31 01:30:18.103426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.103411 5719 scope.go:122] "RemoveContainer" containerID="22278c40cd18f31fdac083ed0ed63c81a63169cf88fbb853880f1201ccfe8c21" Oct 31 01:30:18.211655 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.211614 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq"] Oct 31 01:30:18.315976 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.315945 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf"] Oct 31 01:30:18.320535 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.320510 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-29ece5-29890170-g8vbf"] Oct 31 01:30:18.599944 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.599920 5719 scope.go:122] "RemoveContainer" containerID="545022bb32939c2de3d13b079ed9619631104b05c0060f856701024740d86654" Oct 31 01:30:18.600196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.600174 5719 scope.go:122] "RemoveContainer" containerID="81d78cb6c50a5062c263a1b52f2820d364a735692a2fe262b83ffc27c733e3cb" Oct 31 01:30:18.600331 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.600232 5719 scope.go:122] "RemoveContainer" containerID="7da210fedc6236a93f6e497a6256e8fef902202f236491620a563818b880a9ea" Oct 31 01:30:18.603617 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.603599 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b106522-b041-4f35-a565-b479b712000a" path="/var/lib/kubelet/pods/5b106522-b041-4f35-a565-b479b712000a/volumes" Oct 31 01:30:18.604170 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:18.604054 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" path="/var/lib/kubelet/pods/e1750e79-f6b6-47f7-873b-03c0533fcfc9/volumes" Oct 31 01:30:19.102761 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.102724 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" event={"ID":"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff","Type":"ContainerStarted","Data":"763f1357b0f73ac3c40b2800835c742c4788995d9817e689cea9caf27db2c09a"} Oct 31 01:30:19.106003 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.105977 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" event={"ID":"68bb8b6e-8992-4fa2-9542-416da1e21e39","Type":"ContainerStarted","Data":"5f338785543bc063add5876ec441035d1f9bf7ec598c133ed5fc8123fac5f060"} Oct 31 01:30:19.108079 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.108049 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" event={"ID":"c911889e-a973-4356-9ab7-d12b82434283","Type":"ContainerStarted","Data":"6d380e60718de8d513f50768c89ba84e5971baa95f893ef87e6f4c130111a9d8"} Oct 31 01:30:19.297338 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.297316 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:19.359132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.359097 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-maas-api-service-ca\") pod \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\" (UID: \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\") " Oct 31 01:30:19.359321 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.359244 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-kube-api-access-hrqgg\" (UniqueName: \"kubernetes.io/projected/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-kube-api-access-hrqgg\") pod \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\" (UID: \"5e651e26-89f2-4bab-bd11-8291fc0ac2c0\") " Oct 31 01:30:19.359507 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.359483 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-maas-api-service-ca" pod "5e651e26-89f2-4bab-bd11-8291fc0ac2c0" (UID: "5e651e26-89f2-4bab-bd11-8291fc0ac2c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:19.361406 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.361384 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-kube-api-access-hrqgg" pod "5e651e26-89f2-4bab-bd11-8291fc0ac2c0" (UID: "5e651e26-89f2-4bab-bd11-8291fc0ac2c0"). InnerVolumeSpecName "kube-api-access-hrqgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:19.460584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.460528 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrqgg\" (UniqueName: \"kubernetes.io/projected/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-kube-api-access-hrqgg\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:19.460584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:19.460554 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e651e26-89f2-4bab-bd11-8291fc0ac2c0-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:20.119800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.119761 5719 generic.go:356] "Generic (PLEG): container finished" podID="c911889e-a973-4356-9ab7-d12b82434283" containerID="6d380e60718de8d513f50768c89ba84e5971baa95f893ef87e6f4c130111a9d8" exitCode=60 Oct 31 01:30:20.120354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.119831 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" event={"ID":"c911889e-a973-4356-9ab7-d12b82434283","Type":"ContainerDied","Data":"6d380e60718de8d513f50768c89ba84e5971baa95f893ef87e6f4c130111a9d8"} Oct 31 01:30:20.120354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.119889 5719 scope.go:122] "RemoveContainer" containerID="545022bb32939c2de3d13b079ed9619631104b05c0060f856701024740d86654" Oct 31 01:30:20.120354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.120347 5719 scope.go:122] "RemoveContainer" containerID="6d380e60718de8d513f50768c89ba84e5971baa95f893ef87e6f4c130111a9d8" Oct 31 01:30:20.120671 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:20.120627 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf_odh-ai-gateway-infra(c911889e-a973-4356-9ab7-d12b82434283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" podUID="c911889e-a973-4356-9ab7-d12b82434283" Oct 31 01:30:20.121737 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.121716 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" Oct 31 01:30:20.121877 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.121741 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq" event={"ID":"5e651e26-89f2-4bab-bd11-8291fc0ac2c0","Type":"ContainerDied","Data":"f816b642004bf744e286d1f2e8da70dd0d81753f3a785d97d1baa90a3ea90a27"} Oct 31 01:30:20.124962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.124938 5719 generic.go:356] "Generic (PLEG): container finished" podID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerID="763f1357b0f73ac3c40b2800835c742c4788995d9817e689cea9caf27db2c09a" exitCode=60 Oct 31 01:30:20.125091 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.125027 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" event={"ID":"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff","Type":"ContainerDied","Data":"763f1357b0f73ac3c40b2800835c742c4788995d9817e689cea9caf27db2c09a"} Oct 31 01:30:20.125362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.125345 5719 scope.go:122] "RemoveContainer" containerID="763f1357b0f73ac3c40b2800835c742c4788995d9817e689cea9caf27db2c09a" Oct 31 01:30:20.125572 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:20.125552 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7_odh-ai-gateway-infra(7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" Oct 31 01:30:20.127210 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.127192 5719 generic.go:356] "Generic (PLEG): container finished" podID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerID="5f338785543bc063add5876ec441035d1f9bf7ec598c133ed5fc8123fac5f060" exitCode=60 Oct 31 01:30:20.127317 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.127262 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" event={"ID":"68bb8b6e-8992-4fa2-9542-416da1e21e39","Type":"ContainerDied","Data":"5f338785543bc063add5876ec441035d1f9bf7ec598c133ed5fc8123fac5f060"} Oct 31 01:30:20.127598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.127581 5719 scope.go:122] "RemoveContainer" containerID="5f338785543bc063add5876ec441035d1f9bf7ec598c133ed5fc8123fac5f060" Oct 31 01:30:20.127862 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:20.127838 5719 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-c95sv_odh-ai-gateway-infra(68bb8b6e-8992-4fa2-9542-416da1e21e39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" Oct 31 01:30:20.135287 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.135253 5719 scope.go:122] "RemoveContainer" containerID="b3b86c436d4d5c48dc7483af246c57cf027b2202dcf2800558d50dfb1743029f" Oct 31 01:30:20.145445 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.145419 5719 scope.go:122] "RemoveContainer" containerID="81d78cb6c50a5062c263a1b52f2820d364a735692a2fe262b83ffc27c733e3cb" Oct 31 01:30:20.157721 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.157678 5719 scope.go:122] "RemoveContainer" containerID="7da210fedc6236a93f6e497a6256e8fef902202f236491620a563818b880a9ea" Oct 31 01:30:20.233299 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.233273 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7"] Oct 31 01:30:20.345853 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.345821 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv"] Oct 31 01:30:20.453398 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.453332 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq"] Oct 31 01:30:20.457794 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.457769 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cb9a07-29890170-cd2tq"] Oct 31 01:30:20.570140 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.570114 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf"] Oct 31 01:30:20.602500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:20.602474 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" path="/var/lib/kubelet/pods/5e651e26-89f2-4bab-bd11-8291fc0ac2c0/volumes" Oct 31 01:30:21.305352 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.305258 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:21.375822 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.375797 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:21.379331 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.379306 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:21.388885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.388838 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-kube-api-access-thl9b\" (UniqueName: \"kubernetes.io/projected/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-kube-api-access-thl9b\") pod \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\" (UID: \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\") " Oct 31 01:30:21.388885 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.388877 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-maas-api-service-ca\") pod \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\" (UID: \"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff\") " Oct 31 01:30:21.389287 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.389265 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-maas-api-service-ca" pod "7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" (UID: "7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:21.389369 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.389356 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.390983 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.390964 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-kube-api-access-thl9b" pod "7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" (UID: "7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff"). InnerVolumeSpecName "kube-api-access-thl9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:21.490198 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.490169 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68bb8b6e-8992-4fa2-9542-416da1e21e39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bb8b6e-8992-4fa2-9542-416da1e21e39-maas-api-service-ca\") pod \"68bb8b6e-8992-4fa2-9542-416da1e21e39\" (UID: \"68bb8b6e-8992-4fa2-9542-416da1e21e39\") " Oct 31 01:30:21.490357 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.490236 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c911889e-a973-4356-9ab7-d12b82434283-kube-api-access-lpd7f\" (UniqueName: \"kubernetes.io/projected/c911889e-a973-4356-9ab7-d12b82434283-kube-api-access-lpd7f\") pod \"c911889e-a973-4356-9ab7-d12b82434283\" (UID: \"c911889e-a973-4356-9ab7-d12b82434283\") " Oct 31 01:30:21.490416 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.490372 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bb8b6e-8992-4fa2-9542-416da1e21e39-kube-api-access-9r4vr\" (UniqueName: \"kubernetes.io/projected/68bb8b6e-8992-4fa2-9542-416da1e21e39-kube-api-access-9r4vr\") pod \"68bb8b6e-8992-4fa2-9542-416da1e21e39\" (UID: \"68bb8b6e-8992-4fa2-9542-416da1e21e39\") " Oct 31 01:30:21.490477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.490431 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c911889e-a973-4356-9ab7-d12b82434283-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911889e-a973-4356-9ab7-d12b82434283-maas-api-service-ca\") pod \"c911889e-a973-4356-9ab7-d12b82434283\" (UID: \"c911889e-a973-4356-9ab7-d12b82434283\") " Oct 31 01:30:21.490532 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.490507 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bb8b6e-8992-4fa2-9542-416da1e21e39-maas-api-service-ca" pod "68bb8b6e-8992-4fa2-9542-416da1e21e39" (UID: "68bb8b6e-8992-4fa2-9542-416da1e21e39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:21.490809 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.490787 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68bb8b6e-8992-4fa2-9542-416da1e21e39-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.490884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.490814 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thl9b\" (UniqueName: \"kubernetes.io/projected/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff-kube-api-access-thl9b\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.491030 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.491001 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c911889e-a973-4356-9ab7-d12b82434283-maas-api-service-ca" pod "c911889e-a973-4356-9ab7-d12b82434283" (UID: "c911889e-a973-4356-9ab7-d12b82434283"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:21.492925 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.492896 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bb8b6e-8992-4fa2-9542-416da1e21e39-kube-api-access-9r4vr" pod "68bb8b6e-8992-4fa2-9542-416da1e21e39" (UID: "68bb8b6e-8992-4fa2-9542-416da1e21e39"). InnerVolumeSpecName "kube-api-access-9r4vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:21.493009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.492994 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c911889e-a973-4356-9ab7-d12b82434283-kube-api-access-lpd7f" pod "c911889e-a973-4356-9ab7-d12b82434283" (UID: "c911889e-a973-4356-9ab7-d12b82434283"). InnerVolumeSpecName "kube-api-access-lpd7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:21.591685 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.591652 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c911889e-a973-4356-9ab7-d12b82434283-maas-api-service-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.591867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.591713 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpd7f\" (UniqueName: \"kubernetes.io/projected/c911889e-a973-4356-9ab7-d12b82434283-kube-api-access-lpd7f\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.591867 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:21.591730 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r4vr\" (UniqueName: \"kubernetes.io/projected/68bb8b6e-8992-4fa2-9542-416da1e21e39-kube-api-access-9r4vr\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:22.021568 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.021509 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:30:22.096290 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.096262 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176128e0-a162-4957-8fc3-aba7a4076be7-kube-api-access-qz968\" (UniqueName: \"kubernetes.io/projected/176128e0-a162-4957-8fc3-aba7a4076be7-kube-api-access-qz968\") pod \"176128e0-a162-4957-8fc3-aba7a4076be7\" (UID: \"176128e0-a162-4957-8fc3-aba7a4076be7\") " Oct 31 01:30:22.096463 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.096355 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176128e0-a162-4957-8fc3-aba7a4076be7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176128e0-a162-4957-8fc3-aba7a4076be7-plugins-config-volume\") pod \"176128e0-a162-4957-8fc3-aba7a4076be7\" (UID: \"176128e0-a162-4957-8fc3-aba7a4076be7\") " Oct 31 01:30:22.096728 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.096673 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176128e0-a162-4957-8fc3-aba7a4076be7-plugins-config-volume" pod "176128e0-a162-4957-8fc3-aba7a4076be7" (UID: "176128e0-a162-4957-8fc3-aba7a4076be7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:22.096879 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.096862 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/176128e0-a162-4957-8fc3-aba7a4076be7-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:22.098644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.098618 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176128e0-a162-4957-8fc3-aba7a4076be7-kube-api-access-qz968" pod "176128e0-a162-4957-8fc3-aba7a4076be7" (UID: "176128e0-a162-4957-8fc3-aba7a4076be7"). InnerVolumeSpecName "kube-api-access-qz968". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:22.116884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.116864 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:30:22.147357 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.147335 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" Oct 31 01:30:22.147481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.147375 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf" event={"ID":"c911889e-a973-4356-9ab7-d12b82434283","Type":"ContainerDied","Data":"85f547c22c670438b2926579de5108dde55c1614a2040f5005b6c71c31142476"} Oct 31 01:30:22.147481 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.147433 5719 scope.go:122] "RemoveContainer" containerID="6d380e60718de8d513f50768c89ba84e5971baa95f893ef87e6f4c130111a9d8" Oct 31 01:30:22.149036 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.148998 5719 generic.go:356] "Generic (PLEG): container finished" podID="19a35525-1b77-4772-b089-b8a786ab452b" containerID="b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29" exitCode=0 Oct 31 01:30:22.149132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.149093 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" event={"ID":"19a35525-1b77-4772-b089-b8a786ab452b","Type":"ContainerDied","Data":"b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29"} Oct 31 01:30:22.149182 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.149136 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" event={"ID":"19a35525-1b77-4772-b089-b8a786ab452b","Type":"ContainerDied","Data":"9368c24872ef7e36a1d66dc701017472425181b53eb67478d2ec1b4c46a56a01"} Oct 31 01:30:22.149224 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.149192 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l" Oct 31 01:30:22.150752 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.150724 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" event={"ID":"7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff","Type":"ContainerDied","Data":"babba0eecd9b6749a3e5d6b911180e6a62c0f2bf1181dfbcadcb6c0d516c038f"} Oct 31 01:30:22.150889 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.150749 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7" Oct 31 01:30:22.153022 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.152999 5719 generic.go:356] "Generic (PLEG): container finished" podID="176128e0-a162-4957-8fc3-aba7a4076be7" containerID="02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232" exitCode=0 Oct 31 01:30:22.153134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.153040 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" Oct 31 01:30:22.153134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.153089 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" event={"ID":"176128e0-a162-4957-8fc3-aba7a4076be7","Type":"ContainerDied","Data":"02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232"} Oct 31 01:30:22.153134 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.153125 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782" event={"ID":"176128e0-a162-4957-8fc3-aba7a4076be7","Type":"ContainerDied","Data":"a224d54985f5599f42ed9c45affeca70bd7828a1e19454a5727664954c106fa8"} Oct 31 01:30:22.154560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.154536 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" event={"ID":"68bb8b6e-8992-4fa2-9542-416da1e21e39","Type":"ContainerDied","Data":"8c50cb40b394dfc80b666116fda46e3e1f02aa118e382fe9ea61bf52356d62f3"} Oct 31 01:30:22.154686 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.154564 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv" Oct 31 01:30:22.161001 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.160984 5719 scope.go:122] "RemoveContainer" containerID="b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29" Oct 31 01:30:22.172248 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.172225 5719 scope.go:122] "RemoveContainer" containerID="b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29" Oct 31 01:30:22.172516 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:22.172498 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29\": container with ID starting with b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29 not found: ID does not exist" containerID="b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29" Oct 31 01:30:22.172597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.172529 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29"} err="failed to get container status \"b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29\": rpc error: code = NotFound desc = could not find container \"b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29\": container with ID starting with b04c5482177e9deaaf13eb11f10c75d3836f5c8caf7322a0ad7a0f7655c67e29 not found: ID does not exist" Oct 31 01:30:22.172597 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.172552 5719 scope.go:122] "RemoveContainer" containerID="763f1357b0f73ac3c40b2800835c742c4788995d9817e689cea9caf27db2c09a" Oct 31 01:30:22.180558 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.180542 5719 scope.go:122] "RemoveContainer" containerID="02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232" Oct 31 01:30:22.188882 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.188857 5719 scope.go:122] "RemoveContainer" containerID="02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232" Oct 31 01:30:22.189163 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:22.189140 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232\": container with ID starting with 02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232 not found: ID does not exist" containerID="02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232" Oct 31 01:30:22.189240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.189170 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232"} err="failed to get container status \"02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232\": rpc error: code = NotFound desc = could not find container \"02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232\": container with ID starting with 02cafe26e76d57ffe889c1232175c379a4beb7a8b0e2cead2c4b04a42f7d4232 not found: ID does not exist" Oct 31 01:30:22.189240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.189185 5719 scope.go:122] "RemoveContainer" containerID="5f338785543bc063add5876ec441035d1f9bf7ec598c133ed5fc8123fac5f060" Oct 31 01:30:22.197240 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.197224 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a35525-1b77-4772-b089-b8a786ab452b-kube-api-access-fwddg\" (UniqueName: \"kubernetes.io/projected/19a35525-1b77-4772-b089-b8a786ab452b-kube-api-access-fwddg\") pod \"19a35525-1b77-4772-b089-b8a786ab452b\" (UID: \"19a35525-1b77-4772-b089-b8a786ab452b\") " Oct 31 01:30:22.197295 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.197284 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a35525-1b77-4772-b089-b8a786ab452b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a35525-1b77-4772-b089-b8a786ab452b-plugins-config-volume\") pod \"19a35525-1b77-4772-b089-b8a786ab452b\" (UID: \"19a35525-1b77-4772-b089-b8a786ab452b\") " Oct 31 01:30:22.197679 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.197658 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a35525-1b77-4772-b089-b8a786ab452b-plugins-config-volume" pod "19a35525-1b77-4772-b089-b8a786ab452b" (UID: "19a35525-1b77-4772-b089-b8a786ab452b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:22.197785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.197775 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19a35525-1b77-4772-b089-b8a786ab452b-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:22.197828 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.197789 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz968\" (UniqueName: \"kubernetes.io/projected/176128e0-a162-4957-8fc3-aba7a4076be7-kube-api-access-qz968\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:22.199268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.199247 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a35525-1b77-4772-b089-b8a786ab452b-kube-api-access-fwddg" pod "19a35525-1b77-4772-b089-b8a786ab452b" (UID: "19a35525-1b77-4772-b089-b8a786ab452b"). InnerVolumeSpecName "kube-api-access-fwddg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:22.298652 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.298615 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwddg\" (UniqueName: \"kubernetes.io/projected/19a35525-1b77-4772-b089-b8a786ab452b-kube-api-access-fwddg\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:22.351880 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.351859 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf"] Oct 31 01:30:22.353986 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.353968 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-a72754-29890170-hnhgf"] Oct 31 01:30:22.458416 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.458399 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv"] Oct 31 01:30:22.464200 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.464180 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-c95sv"] Oct 31 01:30:22.601555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.601530 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" path="/var/lib/kubelet/pods/68bb8b6e-8992-4fa2-9542-416da1e21e39/volumes" Oct 31 01:30:22.601924 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.601911 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c911889e-a973-4356-9ab7-d12b82434283" path="/var/lib/kubelet/pods/c911889e-a973-4356-9ab7-d12b82434283/volumes" Oct 31 01:30:22.620316 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.620291 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782"] Oct 31 01:30:22.624055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.624033 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a72754-747f45dbc5-sl782"] Oct 31 01:30:22.730054 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.730023 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7"] Oct 31 01:30:22.734882 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.734863 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7f8596-29890170-4gfr7"] Oct 31 01:30:22.839708 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.839670 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l"] Oct 31 01:30:22.845174 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:22.845155 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-a72754-6f55ccb6f5-b2n7l"] Oct 31 01:30:24.605945 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:24.605654 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176128e0-a162-4957-8fc3-aba7a4076be7" path="/var/lib/kubelet/pods/176128e0-a162-4957-8fc3-aba7a4076be7/volumes" Oct 31 01:30:24.606420 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:24.606338 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a35525-1b77-4772-b089-b8a786ab452b" path="/var/lib/kubelet/pods/19a35525-1b77-4772-b089-b8a786ab452b/volumes" Oct 31 01:30:24.611320 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:24.611258 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" path="/var/lib/kubelet/pods/7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff/volumes" Oct 31 01:30:27.496337 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:27.496302 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57"] Oct 31 01:30:27.544145 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:27.544113 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-a72754-vdz57"] Oct 31 01:30:28.602903 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:28.602867 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3e65eb-259c-44d6-8711-be9fc63744f0" path="/var/lib/kubelet/pods/1c3e65eb-259c-44d6-8711-be9fc63744f0/volumes" Oct 31 01:30:33.010069 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:33.010037 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:30:36.310533 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.310498 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr"] Oct 31 01:30:36.311512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311484 5719 cpu_manager.go:414] "containerMap: removing container" podUID="176128e0-a162-4957-8fc3-aba7a4076be7" containerName="payload-processing" Oct 31 01:30:36.311512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311513 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176128e0-a162-4957-8fc3-aba7a4076be7" containerName="payload-processing" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311535 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311542 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311556 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311564 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311577 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311585 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311599 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311605 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311618 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311625 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311637 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311644 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311654 5719 cpu_manager.go:414] "containerMap: removing container" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311660 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311671 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311678 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311721 5719 cpu_manager.go:414] "containerMap: removing container" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311729 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311742 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311748 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311768 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:36.311769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311775 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311786 5719 cpu_manager.go:414] "containerMap: removing container" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311793 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311802 5719 cpu_manager.go:414] "containerMap: removing container" podUID="19a35525-1b77-4772-b089-b8a786ab452b" containerName="payload-pre-processing" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311810 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a35525-1b77-4772-b089-b8a786ab452b" containerName="payload-pre-processing" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311821 5719 cpu_manager.go:414] "containerMap: removing container" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311828 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311840 5719 cpu_manager.go:414] "containerMap: removing container" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311846 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311876 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311885 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311894 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311902 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311914 5719 cpu_manager.go:414] "containerMap: removing container" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311921 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311940 5719 cpu_manager.go:414] "containerMap: removing container" podUID="269c38ce-d974-4242-aa15-21af8c116ddc" containerName="revoke-keys" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.311947 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269c38ce-d974-4242-aa15-21af8c116ddc" containerName="revoke-keys" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312090 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312102 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312111 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312126 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312134 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312143 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312151 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312158 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a35525-1b77-4772-b089-b8a786ab452b" containerName="payload-pre-processing" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312168 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312177 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312185 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7079fd16-ce2b-466b-afb3-fa4a7dd8a7ff" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312193 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bb8b6e-8992-4fa2-9542-416da1e21e39" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312203 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="269c38ce-d974-4242-aa15-21af8c116ddc" containerName="revoke-keys" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312213 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312224 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="176128e0-a162-4957-8fc3-aba7a4076be7" containerName="payload-processing" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312234 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312244 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.312915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.312253 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e651e26-89f2-4bab-bd11-8291fc0ac2c0" containerName="cleanup" Oct 31 01:30:36.317415 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.317396 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.369368 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.369340 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr"] Oct 31 01:30:36.414749 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.414719 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:36.414749 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.414729 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:30:36.414953 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.414785 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:30:36.414953 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.414894 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:30:36.435442 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.435417 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq86n\" (UniqueName: \"kubernetes.io/projected/e9ce634f-edf8-4dd9-9878-c51766b11091-kube-api-access-nq86n\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.435561 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.435544 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.435626 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.435611 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.536981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.536962 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.537073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.537015 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.537167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.537143 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq86n\" (UniqueName: \"kubernetes.io/projected/e9ce634f-edf8-4dd9-9878-c51766b11091-kube-api-access-nq86n\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.537779 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.537757 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.539393 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.539375 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.596033 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.596013 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq86n\" (UniqueName: \"kubernetes.io/projected/e9ce634f-edf8-4dd9-9878-c51766b11091-kube-api-access-nq86n\") pod \"maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:36.630412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:36.630394 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:37.004263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:37.004191 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr"] Oct 31 01:30:37.012387 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:37.012362 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ce634f_edf8_4dd9_9878_c51766b11091.slice/crio-a4a7bc50398aee128a57430b59e761fe0d36d77ffef970e7a862ec3f51b8e20d WatchSource:0}: Error finding container a4a7bc50398aee128a57430b59e761fe0d36d77ffef970e7a862ec3f51b8e20d: Status 404 returned error can't find the container with id a4a7bc50398aee128a57430b59e761fe0d36d77ffef970e7a862ec3f51b8e20d Oct 31 01:30:37.253538 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:37.253506 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" event={"ID":"e9ce634f-edf8-4dd9-9878-c51766b11091","Type":"ContainerStarted","Data":"a4a7bc50398aee128a57430b59e761fe0d36d77ffef970e7a862ec3f51b8e20d"} Oct 31 01:30:38.260967 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:38.260886 5719 generic.go:356] "Generic (PLEG): container finished" podID="e9ce634f-edf8-4dd9-9878-c51766b11091" containerID="c35fbc86c930cceba67d185c6262d85bd39ac02c0a65ec5bef6d0e3be9f78cb4" exitCode=0 Oct 31 01:30:38.261321 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:38.260976 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" event={"ID":"e9ce634f-edf8-4dd9-9878-c51766b11091","Type":"ContainerDied","Data":"c35fbc86c930cceba67d185c6262d85bd39ac02c0a65ec5bef6d0e3be9f78cb4"} Oct 31 01:30:39.404403 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.404371 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:39.458154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.458126 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-xks-ca\") pod \"e9ce634f-edf8-4dd9-9878-c51766b11091\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " Oct 31 01:30:39.458253 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.458159 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ce634f-edf8-4dd9-9878-c51766b11091-kube-api-access-nq86n\" (UniqueName: \"kubernetes.io/projected/e9ce634f-edf8-4dd9-9878-c51766b11091-kube-api-access-nq86n\") pod \"e9ce634f-edf8-4dd9-9878-c51766b11091\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " Oct 31 01:30:39.458253 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.458241 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-openshift-ca\") pod \"e9ce634f-edf8-4dd9-9878-c51766b11091\" (UID: \"e9ce634f-edf8-4dd9-9878-c51766b11091\") " Oct 31 01:30:39.458631 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.458606 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-openshift-ca" pod "e9ce634f-edf8-4dd9-9878-c51766b11091" (UID: "e9ce634f-edf8-4dd9-9878-c51766b11091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:39.460228 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.460205 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-xks-ca" pod "e9ce634f-edf8-4dd9-9878-c51766b11091" (UID: "e9ce634f-edf8-4dd9-9878-c51766b11091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:39.460311 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.460298 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ce634f-edf8-4dd9-9878-c51766b11091-kube-api-access-nq86n" pod "e9ce634f-edf8-4dd9-9878-c51766b11091" (UID: "e9ce634f-edf8-4dd9-9878-c51766b11091"). InnerVolumeSpecName "kube-api-access-nq86n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:39.559549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.559488 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:39.559549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.559515 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9ce634f-edf8-4dd9-9878-c51766b11091-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:39.559549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:39.559524 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq86n\" (UniqueName: \"kubernetes.io/projected/e9ce634f-edf8-4dd9-9878-c51766b11091-kube-api-access-nq86n\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:40.275407 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:40.275378 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" Oct 31 01:30:40.275582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:40.275413 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr" event={"ID":"e9ce634f-edf8-4dd9-9878-c51766b11091","Type":"ContainerDied","Data":"a4a7bc50398aee128a57430b59e761fe0d36d77ffef970e7a862ec3f51b8e20d"} Oct 31 01:30:40.275582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:40.275471 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4a7bc50398aee128a57430b59e761fe0d36d77ffef970e7a862ec3f51b8e20d" Oct 31 01:30:40.798530 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:40.798499 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj"] Oct 31 01:30:40.846500 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:40.846474 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84"] Oct 31 01:30:40.848412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:40.848392 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f"] Oct 31 01:30:43.530478 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:43.530442 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:30:48.532477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:48.532447 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6"] Oct 31 01:30:48.579960 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:48.579934 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-29ece5-trml6"] Oct 31 01:30:48.601029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:48.600998 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f702c89-e58f-45ad-91b4-710cbb4cf351" path="/var/lib/kubelet/pods/1f702c89-e58f-45ad-91b4-710cbb4cf351/volumes" Oct 31 01:30:49.713584 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:49.713544 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd"] Oct 31 01:30:49.761115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:49.761076 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b63c24-65mzd"] Oct 31 01:30:50.602081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:50.602048 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc6c5cf-8749-49b4-88dd-6dd6a25c6845" path="/var/lib/kubelet/pods/cfc6c5cf-8749-49b4-88dd-6dd6a25c6845/volumes" Oct 31 01:30:55.866744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:55.866675 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" containerID="cri-o://ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d" gracePeriod=45 Oct 31 01:30:55.867652 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:55.867624 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" containerID="cri-o://05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2" gracePeriod=45 Oct 31 01:30:56.154150 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.154125 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 01:30:56.157289 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.157268 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 01:30:56.203241 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203219 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-dshm\") pod \"06cc3e5d-ce95-4db1-be59-c187270251c3\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " Oct 31 01:30:56.203357 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203264 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06cc3e5d-ce95-4db1-be59-c187270251c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/06cc3e5d-ce95-4db1-be59-c187270251c3-tls-certs\") pod \"06cc3e5d-ce95-4db1-be59-c187270251c3\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " Oct 31 01:30:56.203357 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203296 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06cc3e5d-ce95-4db1-be59-c187270251c3-kube-api-access-rb8n6\" (UniqueName: \"kubernetes.io/projected/06cc3e5d-ce95-4db1-be59-c187270251c3-kube-api-access-rb8n6\") pod \"06cc3e5d-ce95-4db1-be59-c187270251c3\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " Oct 31 01:30:56.203357 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203331 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-home\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-home\") pod \"06cc3e5d-ce95-4db1-be59-c187270251c3\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " Oct 31 01:30:56.203543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203363 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-model-cache\") pod \"06cc3e5d-ce95-4db1-be59-c187270251c3\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " Oct 31 01:30:56.203543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203449 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-dshm\") pod \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " Oct 31 01:30:56.203543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203492 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tmp-dir\") pod \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " Oct 31 01:30:56.203543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203518 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-home\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-home\") pod \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " Oct 31 01:30:56.203810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203557 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-model-cache\") pod \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " Oct 31 01:30:56.203810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203633 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-home" pod "06cc3e5d-ce95-4db1-be59-c187270251c3" (UID: "06cc3e5d-ce95-4db1-be59-c187270251c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.203810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203642 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-model-cache" pod "06cc3e5d-ce95-4db1-be59-c187270251c3" (UID: "06cc3e5d-ce95-4db1-be59-c187270251c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.203810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203662 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6e3e81-a1f3-4266-ae60-ece863ea0416-kube-api-access-c84x5\" (UniqueName: \"kubernetes.io/projected/ba6e3e81-a1f3-4266-ae60-ece863ea0416-kube-api-access-c84x5\") pod \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " Oct 31 01:30:56.203810 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203757 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tls-certs\") pod \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\" (UID: \"ba6e3e81-a1f3-4266-ae60-ece863ea0416\") " Oct 31 01:30:56.204247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203857 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-home" pod "ba6e3e81-a1f3-4266-ae60-ece863ea0416" (UID: "ba6e3e81-a1f3-4266-ae60-ece863ea0416"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.204247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.203866 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-tmp-dir\") pod \"06cc3e5d-ce95-4db1-be59-c187270251c3\" (UID: \"06cc3e5d-ce95-4db1-be59-c187270251c3\") " Oct 31 01:30:56.204247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.204107 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tmp-dir" pod "ba6e3e81-a1f3-4266-ae60-ece863ea0416" (UID: "ba6e3e81-a1f3-4266-ae60-ece863ea0416"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.204247 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.204164 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-model-cache" pod "ba6e3e81-a1f3-4266-ae60-ece863ea0416" (UID: "ba6e3e81-a1f3-4266-ae60-ece863ea0416"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.204536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.204428 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.204536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.204449 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.204536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.204463 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.204536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.204497 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.204536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.204512 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.204843 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.204756 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-tmp-dir" pod "06cc3e5d-ce95-4db1-be59-c187270251c3" (UID: "06cc3e5d-ce95-4db1-be59-c187270251c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.206071 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.206017 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-dshm" pod "06cc3e5d-ce95-4db1-be59-c187270251c3" (UID: "06cc3e5d-ce95-4db1-be59-c187270251c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.206199 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.206112 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06cc3e5d-ce95-4db1-be59-c187270251c3-tls-certs" pod "06cc3e5d-ce95-4db1-be59-c187270251c3" (UID: "06cc3e5d-ce95-4db1-be59-c187270251c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:56.206293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.206268 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6e3e81-a1f3-4266-ae60-ece863ea0416-kube-api-access-c84x5" pod "ba6e3e81-a1f3-4266-ae60-ece863ea0416" (UID: "ba6e3e81-a1f3-4266-ae60-ece863ea0416"). InnerVolumeSpecName "kube-api-access-c84x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:56.206368 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.206289 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cc3e5d-ce95-4db1-be59-c187270251c3-kube-api-access-rb8n6" pod "06cc3e5d-ce95-4db1-be59-c187270251c3" (UID: "06cc3e5d-ce95-4db1-be59-c187270251c3"). InnerVolumeSpecName "kube-api-access-rb8n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:56.206554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.206537 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tls-certs" pod "ba6e3e81-a1f3-4266-ae60-ece863ea0416" (UID: "ba6e3e81-a1f3-4266-ae60-ece863ea0416"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:56.207106 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.207090 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-dshm" pod "ba6e3e81-a1f3-4266-ae60-ece863ea0416" (UID: "ba6e3e81-a1f3-4266-ae60-ece863ea0416"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.301021 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.300985 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" containerID="cri-o://aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570" gracePeriod=45 Oct 31 01:30:56.306073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.305961 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c84x5\" (UniqueName: \"kubernetes.io/projected/ba6e3e81-a1f3-4266-ae60-ece863ea0416-kube-api-access-c84x5\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.306073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.305983 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6e3e81-a1f3-4266-ae60-ece863ea0416-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.306073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.305993 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.306073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.306000 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06cc3e5d-ce95-4db1-be59-c187270251c3-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.306073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.306008 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06cc3e5d-ce95-4db1-be59-c187270251c3-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.306073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.306016 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8n6\" (UniqueName: \"kubernetes.io/projected/06cc3e5d-ce95-4db1-be59-c187270251c3-kube-api-access-rb8n6\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.306073 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.306027 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba6e3e81-a1f3-4266-ae60-ece863ea0416-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.364989 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.364965 5719 generic.go:356] "Generic (PLEG): container finished" podID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerID="ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d" exitCode=0 Oct 31 01:30:56.365112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.365012 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" Oct 31 01:30:56.365112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.365043 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" event={"ID":"06cc3e5d-ce95-4db1-be59-c187270251c3","Type":"ContainerDied","Data":"ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d"} Oct 31 01:30:56.365112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.365080 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84" event={"ID":"06cc3e5d-ce95-4db1-be59-c187270251c3","Type":"ContainerDied","Data":"e651c88e7e75d500de4e729954ede2afc069144d5a2b7135f8ace991fe4d86ed"} Oct 31 01:30:56.365112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.365101 5719 scope.go:122] "RemoveContainer" containerID="ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d" Oct 31 01:30:56.366550 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.366530 5719 generic.go:356] "Generic (PLEG): container finished" podID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerID="05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2" exitCode=0 Oct 31 01:30:56.366631 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.366559 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" Oct 31 01:30:56.366631 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.366607 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" event={"ID":"ba6e3e81-a1f3-4266-ae60-ece863ea0416","Type":"ContainerDied","Data":"05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2"} Oct 31 01:30:56.366768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.366637 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f" event={"ID":"ba6e3e81-a1f3-4266-ae60-ece863ea0416","Type":"ContainerDied","Data":"3860787eb05ebec6b2cef4cae8ca65960e00c0fca58489518b331c9198220598"} Oct 31 01:30:56.415089 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.415060 5719 scope.go:122] "RemoveContainer" containerID="ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d" Oct 31 01:30:56.415473 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:56.415451 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d\": container with ID starting with ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d not found: ID does not exist" containerID="ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d" Oct 31 01:30:56.415592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.415486 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d"} err="failed to get container status \"ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d\": rpc error: code = NotFound desc = could not find container \"ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d\": container with ID starting with ff1905777c62afa61deaf2134dc72647fe3e0838d84dbd5ff4ee40ef9ab3509d not found: ID does not exist" Oct 31 01:30:56.415592 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.415513 5719 scope.go:122] "RemoveContainer" containerID="05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2" Oct 31 01:30:56.427553 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.427522 5719 scope.go:122] "RemoveContainer" containerID="05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2" Oct 31 01:30:56.427888 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:56.427869 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2\": container with ID starting with 05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2 not found: ID does not exist" containerID="05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2" Oct 31 01:30:56.427974 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.427899 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2"} err="failed to get container status \"05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2\": rpc error: code = NotFound desc = could not find container \"05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2\": container with ID starting with 05d2499f1311d65a6c87536143a1073ccbd3fc7c666dd396e5ab5d6884b27dc2 not found: ID does not exist" Oct 31 01:30:56.510064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.510036 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f"] Oct 31 01:30:56.539776 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.539753 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 01:30:56.543265 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.543243 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/simulator-w4-ef2572-kserve-5c87dd56c8-56j9f"] Oct 31 01:30:56.601608 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.601580 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" path="/var/lib/kubelet/pods/ba6e3e81-a1f3-4266-ae60-ece863ea0416/volumes" Oct 31 01:30:56.608425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.608407 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-home\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-home\") pod \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " Oct 31 01:30:56.608497 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.608443 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-kube-api-access-hr87c\" (UniqueName: \"kubernetes.io/projected/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-kube-api-access-hr87c\") pod \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " Oct 31 01:30:56.608544 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.608529 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tmp-dir\") pod \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " Oct 31 01:30:56.608673 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.608656 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-home" pod "b09e9548-a2b8-4a34-ab3e-6a5e791fc063" (UID: "b09e9548-a2b8-4a34-ab3e-6a5e791fc063"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.608745 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.608724 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-dshm\") pod \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " Oct 31 01:30:56.608784 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.608757 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tmp-dir" pod "b09e9548-a2b8-4a34-ab3e-6a5e791fc063" (UID: "b09e9548-a2b8-4a34-ab3e-6a5e791fc063"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.608784 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.608765 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-model-cache\") pod \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " Oct 31 01:30:56.608853 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.608833 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tls-certs\" (UniqueName: \"kubernetes.io/secret/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tls-certs\") pod \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\" (UID: \"b09e9548-a2b8-4a34-ab3e-6a5e791fc063\") " Oct 31 01:30:56.609061 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.609032 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-model-cache" pod "b09e9548-a2b8-4a34-ab3e-6a5e791fc063" (UID: "b09e9548-a2b8-4a34-ab3e-6a5e791fc063"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.609212 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.609196 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.609267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.609217 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.609267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.609231 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.610727 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.610685 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-dshm" pod "b09e9548-a2b8-4a34-ab3e-6a5e791fc063" (UID: "b09e9548-a2b8-4a34-ab3e-6a5e791fc063"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:56.610831 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.610787 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-kube-api-access-hr87c" pod "b09e9548-a2b8-4a34-ab3e-6a5e791fc063" (UID: "b09e9548-a2b8-4a34-ab3e-6a5e791fc063"). InnerVolumeSpecName "kube-api-access-hr87c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:56.611204 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.611178 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tls-certs" pod "b09e9548-a2b8-4a34-ab3e-6a5e791fc063" (UID: "b09e9548-a2b8-4a34-ab3e-6a5e791fc063"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:56.647512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.647464 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84"] Oct 31 01:30:56.651381 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.651364 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/premium-w4-ef2572-kserve-5df897999f-qvk84"] Oct 31 01:30:56.710212 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.710192 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.710212 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.710212 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:56.710330 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:56.710221 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr87c\" (UniqueName: \"kubernetes.io/projected/b09e9548-a2b8-4a34-ab3e-6a5e791fc063-kube-api-access-hr87c\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:30:57.375857 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.375824 5719 generic.go:356] "Generic (PLEG): container finished" podID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerID="aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570" exitCode=0 Oct 31 01:30:57.376301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.375868 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" Oct 31 01:30:57.376301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.375881 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" event={"ID":"b09e9548-a2b8-4a34-ab3e-6a5e791fc063","Type":"ContainerDied","Data":"aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570"} Oct 31 01:30:57.376301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.375911 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj" event={"ID":"b09e9548-a2b8-4a34-ab3e-6a5e791fc063","Type":"ContainerDied","Data":"fb8151c6ace1020b8336f214df34ad81efd1237a125bfc3b8c1774fa33bc6f80"} Oct 31 01:30:57.376301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.375925 5719 scope.go:122] "RemoveContainer" containerID="aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570" Oct 31 01:30:57.385503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.385485 5719 scope.go:122] "RemoveContainer" containerID="aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570" Oct 31 01:30:57.385765 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:30:57.385749 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570\": container with ID starting with aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570 not found: ID does not exist" containerID="aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570" Oct 31 01:30:57.385829 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.385771 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570"} err="failed to get container status \"aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570\": rpc error: code = NotFound desc = could not find container \"aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570\": container with ID starting with aa9cb8ff9b1c8878b94ac034db13d0dc6a463e455ffd23611547998d01ed5570 not found: ID does not exist" Oct 31 01:30:57.541271 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.541240 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj"] Oct 31 01:30:57.543930 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:57.543904 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ef2572/distinct-w4-ef2572-kserve-54cf6ccf4b-tlshj"] Oct 31 01:30:58.600759 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:58.600729 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" path="/var/lib/kubelet/pods/06cc3e5d-ce95-4db1-be59-c187270251c3/volumes" Oct 31 01:30:58.601149 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:58.601122 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" path="/var/lib/kubelet/pods/b09e9548-a2b8-4a34-ab3e-6a5e791fc063/volumes" Oct 31 01:30:59.070926 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.070885 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff"] Oct 31 01:30:59.071404 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071384 5719 cpu_manager.go:414] "containerMap: removing container" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" Oct 31 01:30:59.071506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071405 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" Oct 31 01:30:59.071506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071454 5719 cpu_manager.go:414] "containerMap: removing container" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" Oct 31 01:30:59.071506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071464 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" Oct 31 01:30:59.071506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071483 5719 cpu_manager.go:414] "containerMap: removing container" podUID="e9ce634f-edf8-4dd9-9878-c51766b11091" containerName="revoke-keys" Oct 31 01:30:59.071506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071491 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ce634f-edf8-4dd9-9878-c51766b11091" containerName="revoke-keys" Oct 31 01:30:59.071506 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071506 5719 cpu_manager.go:414] "containerMap: removing container" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071513 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071522 5719 cpu_manager.go:414] "containerMap: removing container" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071529 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071633 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6e3e81-a1f3-4266-ae60-ece863ea0416" containerName="main" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071648 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ce634f-edf8-4dd9-9878-c51766b11091" containerName="revoke-keys" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071659 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09e9548-a2b8-4a34-ab3e-6a5e791fc063" containerName="main" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071670 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b106522-b041-4f35-a565-b479b712000a" containerName="cleanup" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071682 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1750e79-f6b6-47f7-873b-03c0533fcfc9" containerName="cleanup" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071711 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cc3e5d-ce95-4db1-be59-c187270251c3" containerName="main" Oct 31 01:30:59.071868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.071724 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="c911889e-a973-4356-9ab7-d12b82434283" containerName="cleanup" Oct 31 01:30:59.076454 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.076432 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.130276 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.130257 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff"] Oct 31 01:30:59.132886 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.132868 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmpf\" (UniqueName: \"kubernetes.io/projected/6e05e9bf-502a-4690-82d6-9935089983a2-kube-api-access-stmpf\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.133012 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.132921 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.133111 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.133093 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.175816 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.175796 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:30:59.175962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.175826 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:59.177620 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.177595 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s"] Oct 31 01:30:59.178293 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.178274 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:30:59.178523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.178507 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:30:59.181366 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.181348 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.229667 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.229644 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s"] Oct 31 01:30:59.234209 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.234184 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.234357 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.234259 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5r7q\" (UniqueName: \"kubernetes.io/projected/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-kube-api-access-z5r7q\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.234357 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.234327 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.234546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.234431 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.234546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.234481 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.234546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.234531 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmpf\" (UniqueName: \"kubernetes.io/projected/6e05e9bf-502a-4690-82d6-9935089983a2-kube-api-access-stmpf\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.235618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.235600 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.237433 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.237416 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.293485 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.293460 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmpf\" (UniqueName: \"kubernetes.io/projected/6e05e9bf-502a-4690-82d6-9935089983a2-kube-api-access-stmpf\") pod \"maas-api-revoke-sub-f61e8de3-jdqff\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.335051 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.334998 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.335051 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.335040 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.335188 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.335128 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5r7q\" (UniqueName: \"kubernetes.io/projected/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-kube-api-access-z5r7q\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.335557 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.335537 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.337348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.337328 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.389072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.389050 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:30:59.393862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.393839 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5r7q\" (UniqueName: \"kubernetes.io/projected/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-kube-api-access-z5r7q\") pod \"maas-api-revoke-sub-51e96aed-z9x9s\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.494008 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.493977 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:30:59.760428 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.760366 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff"] Oct 31 01:30:59.767174 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:59.767149 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e05e9bf_502a_4690_82d6_9935089983a2.slice/crio-5c7d2956445b1773305181d1370cbba538eea2e5192885ef009111527e57d034 WatchSource:0}: Error finding container 5c7d2956445b1773305181d1370cbba538eea2e5192885ef009111527e57d034: Status 404 returned error can't find the container with id 5c7d2956445b1773305181d1370cbba538eea2e5192885ef009111527e57d034 Oct 31 01:30:59.875827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:30:59.875798 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s"] Oct 31 01:30:59.882913 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:30:59.882889 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28af5f7b_4f8d_4a73_b93b_e223ce8ebe7b.slice/crio-05c30814c9422455075dbc866331babe3f7699b22b3e591b03494df4098d7754 WatchSource:0}: Error finding container 05c30814c9422455075dbc866331babe3f7699b22b3e591b03494df4098d7754: Status 404 returned error can't find the container with id 05c30814c9422455075dbc866331babe3f7699b22b3e591b03494df4098d7754 Oct 31 01:31:00.396614 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:00.396579 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" event={"ID":"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b","Type":"ContainerStarted","Data":"aada5586ec1eb816576030ef1b879644bd79d820dba193cd724e710837273812"} Oct 31 01:31:00.396780 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:00.396674 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" event={"ID":"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b","Type":"ContainerStarted","Data":"05c30814c9422455075dbc866331babe3f7699b22b3e591b03494df4098d7754"} Oct 31 01:31:00.398117 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:00.398090 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" event={"ID":"6e05e9bf-502a-4690-82d6-9935089983a2","Type":"ContainerStarted","Data":"38a1f14a16c10ffe682092b9a370765f6d8b263cedfa3234523c6e802645cfaa"} Oct 31 01:31:00.398225 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:00.398137 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" event={"ID":"6e05e9bf-502a-4690-82d6-9935089983a2","Type":"ContainerStarted","Data":"5c7d2956445b1773305181d1370cbba538eea2e5192885ef009111527e57d034"} Oct 31 01:31:00.508685 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:00.508644 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" podStartSLOduration=1.508631099 podStartE2EDuration="1.508631099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:00.507678509 +0000 UTC m=+3742.043820815" watchObservedRunningTime="2026-10-31 01:31:00.508631099 +0000 UTC m=+3742.044773398" Oct 31 01:31:00.618997 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:00.618952 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" podStartSLOduration=1.618939288 podStartE2EDuration="1.618939288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:00.618745922 +0000 UTC m=+3742.154888227" watchObservedRunningTime="2026-10-31 01:31:00.618939288 +0000 UTC m=+3742.155081592" Oct 31 01:31:01.407292 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:01.407262 5719 generic.go:356] "Generic (PLEG): container finished" podID="28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b" containerID="aada5586ec1eb816576030ef1b879644bd79d820dba193cd724e710837273812" exitCode=0 Oct 31 01:31:01.407789 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:01.407342 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" event={"ID":"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b","Type":"ContainerDied","Data":"aada5586ec1eb816576030ef1b879644bd79d820dba193cd724e710837273812"} Oct 31 01:31:01.409029 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:01.409004 5719 generic.go:356] "Generic (PLEG): container finished" podID="6e05e9bf-502a-4690-82d6-9935089983a2" containerID="38a1f14a16c10ffe682092b9a370765f6d8b263cedfa3234523c6e802645cfaa" exitCode=0 Oct 31 01:31:01.409142 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:01.409059 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" event={"ID":"6e05e9bf-502a-4690-82d6-9935089983a2","Type":"ContainerDied","Data":"38a1f14a16c10ffe682092b9a370765f6d8b263cedfa3234523c6e802645cfaa"} Oct 31 01:31:02.568127 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.568101 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:31:02.571147 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.571126 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:31:02.668787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.668734 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-openshift-ca\") pod \"6e05e9bf-502a-4690-82d6-9935089983a2\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " Oct 31 01:31:02.668787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.668775 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-kube-api-access-z5r7q\" (UniqueName: \"kubernetes.io/projected/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-kube-api-access-z5r7q\") pod \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " Oct 31 01:31:02.668924 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.668848 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-xks-ca\") pod \"6e05e9bf-502a-4690-82d6-9935089983a2\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " Oct 31 01:31:02.668924 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.668887 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e05e9bf-502a-4690-82d6-9935089983a2-kube-api-access-stmpf\" (UniqueName: \"kubernetes.io/projected/6e05e9bf-502a-4690-82d6-9935089983a2-kube-api-access-stmpf\") pod \"6e05e9bf-502a-4690-82d6-9935089983a2\" (UID: \"6e05e9bf-502a-4690-82d6-9935089983a2\") " Oct 31 01:31:02.668924 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.668912 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-xks-ca\") pod \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " Oct 31 01:31:02.669033 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.668955 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-openshift-ca\") pod \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\" (UID: \"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b\") " Oct 31 01:31:02.669107 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.669083 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-openshift-ca" pod "6e05e9bf-502a-4690-82d6-9935089983a2" (UID: "6e05e9bf-502a-4690-82d6-9935089983a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:02.669395 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.669373 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:02.669488 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.669421 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-openshift-ca" pod "28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b" (UID: "28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:02.671071 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.671046 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-xks-ca" pod "6e05e9bf-502a-4690-82d6-9935089983a2" (UID: "6e05e9bf-502a-4690-82d6-9935089983a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:31:02.671155 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.671121 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e05e9bf-502a-4690-82d6-9935089983a2-kube-api-access-stmpf" pod "6e05e9bf-502a-4690-82d6-9935089983a2" (UID: "6e05e9bf-502a-4690-82d6-9935089983a2"). InnerVolumeSpecName "kube-api-access-stmpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:02.671502 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.671480 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-xks-ca" pod "28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b" (UID: "28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:31:02.671542 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.671496 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-kube-api-access-z5r7q" pod "28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b" (UID: "28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b"). InnerVolumeSpecName "kube-api-access-z5r7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:02.770157 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.770130 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stmpf\" (UniqueName: \"kubernetes.io/projected/6e05e9bf-502a-4690-82d6-9935089983a2-kube-api-access-stmpf\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:02.770157 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.770153 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:02.770313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.770162 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:02.770313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.770171 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5r7q\" (UniqueName: \"kubernetes.io/projected/28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b-kube-api-access-z5r7q\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:02.770313 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:02.770180 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e05e9bf-502a-4690-82d6-9935089983a2-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:03.008944 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:31:03.008866 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:31:03.422052 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:03.422023 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" Oct 31 01:31:03.422232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:03.422070 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f61e8de3-jdqff" event={"ID":"6e05e9bf-502a-4690-82d6-9935089983a2","Type":"ContainerDied","Data":"5c7d2956445b1773305181d1370cbba538eea2e5192885ef009111527e57d034"} Oct 31 01:31:03.422232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:03.422123 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c7d2956445b1773305181d1370cbba538eea2e5192885ef009111527e57d034" Oct 31 01:31:03.423600 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:03.423577 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" event={"ID":"28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b","Type":"ContainerDied","Data":"05c30814c9422455075dbc866331babe3f7699b22b3e591b03494df4098d7754"} Oct 31 01:31:03.423600 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:03.423592 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e96aed-z9x9s" Oct 31 01:31:03.423821 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:03.423610 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05c30814c9422455075dbc866331babe3f7699b22b3e591b03494df4098d7754" Oct 31 01:31:23.737414 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:23.737379 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:31:28.550648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.550597 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd"] Oct 31 01:31:28.551090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.551010 5719 cpu_manager.go:414] "containerMap: removing container" podUID="28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b" containerName="revoke-keys" Oct 31 01:31:28.551090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.551023 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b" containerName="revoke-keys" Oct 31 01:31:28.551090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.551049 5719 cpu_manager.go:414] "containerMap: removing container" podUID="6e05e9bf-502a-4690-82d6-9935089983a2" containerName="revoke-keys" Oct 31 01:31:28.551090 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.551054 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e05e9bf-502a-4690-82d6-9935089983a2" containerName="revoke-keys" Oct 31 01:31:28.551234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.551128 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e05e9bf-502a-4690-82d6-9935089983a2" containerName="revoke-keys" Oct 31 01:31:28.551234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.551141 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b" containerName="revoke-keys" Oct 31 01:31:28.554351 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.554336 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.595072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.595053 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.595072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.595081 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.595263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.595122 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7lw\" (UniqueName: \"kubernetes.io/projected/741e4a17-5737-41e5-b281-0e43b6cbd3a6-kube-api-access-gr7lw\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.604727 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.604686 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:31:28.604843 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.604762 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:31:28.652552 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.652522 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:31:28.652552 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.652521 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:31:28.654457 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.654437 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd"] Oct 31 01:31:28.696087 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.696059 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.696185 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.696095 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.696185 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.696155 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7lw\" (UniqueName: \"kubernetes.io/projected/741e4a17-5737-41e5-b281-0e43b6cbd3a6-kube-api-access-gr7lw\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.696793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.696771 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.698557 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.698540 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.762635 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.762612 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7lw\" (UniqueName: \"kubernetes.io/projected/741e4a17-5737-41e5-b281-0e43b6cbd3a6-kube-api-access-gr7lw\") pod \"maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:28.867622 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:28.867601 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:29.279443 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:29.279348 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd"] Oct 31 01:31:29.286380 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:31:29.286348 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741e4a17_5737_41e5_b281_0e43b6cbd3a6.slice/crio-5b87c2a67d45236e5c5e5d659d89a0a15b2e3d110cefcc4f9d99f28859705afd WatchSource:0}: Error finding container 5b87c2a67d45236e5c5e5d659d89a0a15b2e3d110cefcc4f9d99f28859705afd: Status 404 returned error can't find the container with id 5b87c2a67d45236e5c5e5d659d89a0a15b2e3d110cefcc4f9d99f28859705afd Oct 31 01:31:29.569087 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:29.569058 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" event={"ID":"741e4a17-5737-41e5-b281-0e43b6cbd3a6","Type":"ContainerStarted","Data":"5b87c2a67d45236e5c5e5d659d89a0a15b2e3d110cefcc4f9d99f28859705afd"} Oct 31 01:31:30.576409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:30.576383 5719 generic.go:356] "Generic (PLEG): container finished" podID="741e4a17-5737-41e5-b281-0e43b6cbd3a6" containerID="a471945da654bc2d2ed1f7141bb9dee0ec63f02df24131748d59ebbe189e20b4" exitCode=0 Oct 31 01:31:30.576785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:30.576419 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" event={"ID":"741e4a17-5737-41e5-b281-0e43b6cbd3a6","Type":"ContainerDied","Data":"a471945da654bc2d2ed1f7141bb9dee0ec63f02df24131748d59ebbe189e20b4"} Oct 31 01:31:31.182737 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.182705 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr"] Oct 31 01:31:31.184723 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.184685 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ef2572-tmksr"] Oct 31 01:31:31.717989 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.717963 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:31.831847 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.831820 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/741e4a17-5737-41e5-b281-0e43b6cbd3a6-kube-api-access-gr7lw\" (UniqueName: \"kubernetes.io/projected/741e4a17-5737-41e5-b281-0e43b6cbd3a6-kube-api-access-gr7lw\") pod \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " Oct 31 01:31:31.831985 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.831933 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-xks-ca\") pod \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " Oct 31 01:31:31.831985 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.831972 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-openshift-ca\") pod \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\" (UID: \"741e4a17-5737-41e5-b281-0e43b6cbd3a6\") " Oct 31 01:31:31.832533 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.832489 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-openshift-ca" pod "741e4a17-5737-41e5-b281-0e43b6cbd3a6" (UID: "741e4a17-5737-41e5-b281-0e43b6cbd3a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:31.834738 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.834650 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-xks-ca" pod "741e4a17-5737-41e5-b281-0e43b6cbd3a6" (UID: "741e4a17-5737-41e5-b281-0e43b6cbd3a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:31:31.834846 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.834805 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/741e4a17-5737-41e5-b281-0e43b6cbd3a6-kube-api-access-gr7lw" pod "741e4a17-5737-41e5-b281-0e43b6cbd3a6" (UID: "741e4a17-5737-41e5-b281-0e43b6cbd3a6"). InnerVolumeSpecName "kube-api-access-gr7lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:31.933554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.933529 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr7lw\" (UniqueName: \"kubernetes.io/projected/741e4a17-5737-41e5-b281-0e43b6cbd3a6-kube-api-access-gr7lw\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:31.933675 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.933556 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:31.933675 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:31.933566 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/741e4a17-5737-41e5-b281-0e43b6cbd3a6-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:32.589600 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:32.589571 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" Oct 31 01:31:32.589784 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:32.589566 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd" event={"ID":"741e4a17-5737-41e5-b281-0e43b6cbd3a6","Type":"ContainerDied","Data":"5b87c2a67d45236e5c5e5d659d89a0a15b2e3d110cefcc4f9d99f28859705afd"} Oct 31 01:31:32.589784 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:32.589662 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b87c2a67d45236e5c5e5d659d89a0a15b2e3d110cefcc4f9d99f28859705afd" Oct 31 01:31:32.600839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:32.600817 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ce634f-edf8-4dd9-9878-c51766b11091" path="/var/lib/kubelet/pods/e9ce634f-edf8-4dd9-9878-c51766b11091/volumes" Oct 31 01:31:33.009452 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:31:33.009378 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:31:38.253767 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:38.253731 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7"] Oct 31 01:31:38.254175 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:38.253964 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" containerID="cri-o://af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c" gracePeriod=30 Oct 31 01:31:42.306151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:42.306119 5719 scope.go:122] "RemoveContainer" containerID="665e7dbe24b8c1df96fafcad33d20d2fbafdf5ffcab5d29ba632e0f1e7a7458d" Oct 31 01:31:42.315727 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:42.315686 5719 scope.go:122] "RemoveContainer" containerID="5ff6c5b96fc0c58dc90900a3db1af078b59c26c20ba8b7e000a91b66b01c0ac3" Oct 31 01:31:42.323768 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:42.323751 5719 scope.go:122] "RemoveContainer" containerID="f29f99bf11858cc3058c1ed39792f9b4507a3b6c86bcdb9000f6ca336842f832" Oct 31 01:31:43.504917 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.504895 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 01:31:43.632062 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632036 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-envoy\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632085 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-certs\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632123 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9973703c-628f-40b6-9aae-df327ba8ad65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9973703c-628f-40b6-9aae-df327ba8ad65-istio-podinfo\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632148 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-kube-api-access-kcvbk\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-kube-api-access-kcvbk\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632192 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632181 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9973703c-628f-40b6-9aae-df327ba8ad65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9973703c-628f-40b6-9aae-df327ba8ad65-istiod-ca-cert\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632217 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-credential-socket\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632269 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-istio-token\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-istio-token\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632315 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-socket\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632436 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-certs" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:43.632601 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632521 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-data\") pod \"9973703c-628f-40b6-9aae-df327ba8ad65\" (UID: \"9973703c-628f-40b6-9aae-df327ba8ad65\") " Oct 31 01:31:43.632739 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632635 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9973703c-628f-40b6-9aae-df327ba8ad65-istiod-ca-cert" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:43.632893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.632870 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-data" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:43.633084 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.633065 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-credential-socket" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:43.633169 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.633104 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-socket" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:43.633234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.633180 5719 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.633234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.633200 5719 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9973703c-628f-40b6-9aae-df327ba8ad65-istiod-ca-cert\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.633234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.633216 5719 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-credential-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.633234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.633230 5719 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-data\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.634435 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.634417 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-envoy" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:43.634512 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.634432 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9973703c-628f-40b6-9aae-df327ba8ad65-istio-podinfo" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:31:43.635004 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.634984 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-istio-token" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:43.635055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.635017 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-kube-api-access-kcvbk" pod "9973703c-628f-40b6-9aae-df327ba8ad65" (UID: "9973703c-628f-40b6-9aae-df327ba8ad65"). InnerVolumeSpecName "kube-api-access-kcvbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:43.653551 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.653527 5719 generic.go:356] "Generic (PLEG): container finished" podID="9973703c-628f-40b6-9aae-df327ba8ad65" containerID="af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c" exitCode=0 Oct 31 01:31:43.653630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.653568 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" Oct 31 01:31:43.653630 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.653604 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" event={"ID":"9973703c-628f-40b6-9aae-df327ba8ad65","Type":"ContainerDied","Data":"af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c"} Oct 31 01:31:43.653744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.653653 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7" event={"ID":"9973703c-628f-40b6-9aae-df327ba8ad65","Type":"ContainerDied","Data":"e26ecfabc54b0753e3714a9d745245f1b5f8ddc2ccb2dac2cd26677613a546d3"} Oct 31 01:31:43.653744 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.653674 5719 scope.go:122] "RemoveContainer" containerID="af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c" Oct 31 01:31:43.670294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.670266 5719 scope.go:122] "RemoveContainer" containerID="af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c" Oct 31 01:31:43.670631 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:31:43.670591 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c\": container with ID starting with af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c not found: ID does not exist" containerID="af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c" Oct 31 01:31:43.670754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.670631 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c"} err="failed to get container status \"af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c\": rpc error: code = NotFound desc = could not find container \"af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c\": container with ID starting with af9852a2358e40517fd9672c3ced0c664defc2b8cd0f4eaaf8920a48338a4c7c not found: ID does not exist" Oct 31 01:31:43.734547 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.734521 5719 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-istio-envoy\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.734547 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.734542 5719 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9973703c-628f-40b6-9aae-df327ba8ad65-istio-podinfo\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.734684 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.734552 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcvbk\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-kube-api-access-kcvbk\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.734684 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.734561 5719 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9973703c-628f-40b6-9aae-df327ba8ad65-istio-token\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.734684 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.734570 5719 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9973703c-628f-40b6-9aae-df327ba8ad65-workload-socket\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:43.776425 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.776400 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7"] Oct 31 01:31:43.780724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:43.780687 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ef2572-openshift-default-7649dd87-d9zt7"] Oct 31 01:31:44.600763 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:44.600729 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" path="/var/lib/kubelet/pods/9973703c-628f-40b6-9aae-df327ba8ad65/volumes" Oct 31 01:31:46.493550 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:46.493476 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:31:47.282099 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:47.282060 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd"] Oct 31 01:31:47.328827 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:47.328798 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c61da9-2llsd"] Oct 31 01:31:48.600902 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:48.600867 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="741e4a17-5737-41e5-b281-0e43b6cbd3a6" path="/var/lib/kubelet/pods/741e4a17-5737-41e5-b281-0e43b6cbd3a6/volumes" Oct 31 01:31:55.517863 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.517825 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps"] Oct 31 01:31:55.518484 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.518462 5719 cpu_manager.go:414] "containerMap: removing container" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" Oct 31 01:31:55.518570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.518486 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" Oct 31 01:31:55.518570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.518518 5719 cpu_manager.go:414] "containerMap: removing container" podUID="741e4a17-5737-41e5-b281-0e43b6cbd3a6" containerName="revoke-keys" Oct 31 01:31:55.518570 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.518527 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="741e4a17-5737-41e5-b281-0e43b6cbd3a6" containerName="revoke-keys" Oct 31 01:31:55.518765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.518655 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="741e4a17-5737-41e5-b281-0e43b6cbd3a6" containerName="revoke-keys" Oct 31 01:31:55.518765 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.518681 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="9973703c-628f-40b6-9aae-df327ba8ad65" containerName="istio-proxy" Oct 31 01:31:55.522623 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.522600 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.622108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.622077 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:31:55.622108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.622097 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:31:55.622108 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.622084 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:31:55.622375 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.622077 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:31:55.624057 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.624033 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps"] Oct 31 01:31:55.635709 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.635672 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.635850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.635763 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.635898 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.635871 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fzr\" (UniqueName: \"kubernetes.io/projected/911a6f16-32c8-44b4-99e1-518471dcc54f-kube-api-access-g8fzr\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.736928 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.736899 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.737044 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.736974 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.737112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.737058 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fzr\" (UniqueName: \"kubernetes.io/projected/911a6f16-32c8-44b4-99e1-518471dcc54f-kube-api-access-g8fzr\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.737687 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.737666 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.739345 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.739317 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.795882 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.795835 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fzr\" (UniqueName: \"kubernetes.io/projected/911a6f16-32c8-44b4-99e1-518471dcc54f-kube-api-access-g8fzr\") pod \"maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:55.838643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:55.838617 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:56.211311 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:56.211280 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps"] Oct 31 01:31:56.218351 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:31:56.218324 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911a6f16_32c8_44b4_99e1_518471dcc54f.slice/crio-c57ec8b0ff7aab0b8816e7517b9fae62de438accd0c0df1496feb9c4bfec9f51 WatchSource:0}: Error finding container c57ec8b0ff7aab0b8816e7517b9fae62de438accd0c0df1496feb9c4bfec9f51: Status 404 returned error can't find the container with id c57ec8b0ff7aab0b8816e7517b9fae62de438accd0c0df1496feb9c4bfec9f51 Oct 31 01:31:56.728048 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:56.728013 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" event={"ID":"911a6f16-32c8-44b4-99e1-518471dcc54f","Type":"ContainerStarted","Data":"74a9a96b29be55eb687fe121c2fbaece1d4dd132133072b6d3c1cced6fc89862"} Oct 31 01:31:56.728409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:56.728072 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" event={"ID":"911a6f16-32c8-44b4-99e1-518471dcc54f","Type":"ContainerStarted","Data":"c57ec8b0ff7aab0b8816e7517b9fae62de438accd0c0df1496feb9c4bfec9f51"} Oct 31 01:31:56.842834 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:56.842789 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" podStartSLOduration=1.842776532 podStartE2EDuration="1.842776532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:56.84175643 +0000 UTC m=+3798.377898736" watchObservedRunningTime="2026-10-31 01:31:56.842776532 +0000 UTC m=+3798.378918836" Oct 31 01:31:57.735137 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:57.735099 5719 generic.go:356] "Generic (PLEG): container finished" podID="911a6f16-32c8-44b4-99e1-518471dcc54f" containerID="74a9a96b29be55eb687fe121c2fbaece1d4dd132133072b6d3c1cced6fc89862" exitCode=0 Oct 31 01:31:57.735555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:57.735172 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" event={"ID":"911a6f16-32c8-44b4-99e1-518471dcc54f","Type":"ContainerDied","Data":"74a9a96b29be55eb687fe121c2fbaece1d4dd132133072b6d3c1cced6fc89862"} Oct 31 01:31:58.872882 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:58.872858 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:58.965543 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:58.965520 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-openshift-ca\") pod \"911a6f16-32c8-44b4-99e1-518471dcc54f\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " Oct 31 01:31:58.965677 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:58.965569 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911a6f16-32c8-44b4-99e1-518471dcc54f-kube-api-access-g8fzr\" (UniqueName: \"kubernetes.io/projected/911a6f16-32c8-44b4-99e1-518471dcc54f-kube-api-access-g8fzr\") pod \"911a6f16-32c8-44b4-99e1-518471dcc54f\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " Oct 31 01:31:58.965677 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:58.965618 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-xks-ca\") pod \"911a6f16-32c8-44b4-99e1-518471dcc54f\" (UID: \"911a6f16-32c8-44b4-99e1-518471dcc54f\") " Oct 31 01:31:58.965875 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:58.965854 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-openshift-ca" pod "911a6f16-32c8-44b4-99e1-518471dcc54f" (UID: "911a6f16-32c8-44b4-99e1-518471dcc54f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:58.965987 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:58.965959 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:58.967830 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:58.967804 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-xks-ca" pod "911a6f16-32c8-44b4-99e1-518471dcc54f" (UID: "911a6f16-32c8-44b4-99e1-518471dcc54f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:31:58.967932 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:58.967835 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911a6f16-32c8-44b4-99e1-518471dcc54f-kube-api-access-g8fzr" pod "911a6f16-32c8-44b4-99e1-518471dcc54f" (UID: "911a6f16-32c8-44b4-99e1-518471dcc54f"). InnerVolumeSpecName "kube-api-access-g8fzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:59.066884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:59.066861 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fzr\" (UniqueName: \"kubernetes.io/projected/911a6f16-32c8-44b4-99e1-518471dcc54f-kube-api-access-g8fzr\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:59.066884 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:59.066883 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911a6f16-32c8-44b4-99e1-518471dcc54f-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:31:59.747368 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:59.747338 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" Oct 31 01:31:59.747368 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:59.747361 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps" event={"ID":"911a6f16-32c8-44b4-99e1-518471dcc54f","Type":"ContainerDied","Data":"c57ec8b0ff7aab0b8816e7517b9fae62de438accd0c0df1496feb9c4bfec9f51"} Oct 31 01:31:59.747575 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:31:59.747403 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57ec8b0ff7aab0b8816e7517b9fae62de438accd0c0df1496feb9c4bfec9f51" Oct 31 01:32:00.767270 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:00.767242 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:32:03.009565 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:32:03.009534 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:32:18.151072 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:18.151038 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps"] Oct 31 01:32:18.198747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:18.198718 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6bd7c8-t8dps"] Oct 31 01:32:18.601932 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:18.601901 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911a6f16-32c8-44b4-99e1-518471dcc54f" path="/var/lib/kubelet/pods/911a6f16-32c8-44b4-99e1-518471dcc54f/volumes" Oct 31 01:32:22.108458 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:22.108425 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:32:26.661853 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.661822 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r"] Oct 31 01:32:26.662445 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.662422 5719 cpu_manager.go:414] "containerMap: removing container" podUID="911a6f16-32c8-44b4-99e1-518471dcc54f" containerName="revoke-keys" Oct 31 01:32:26.662445 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.662442 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911a6f16-32c8-44b4-99e1-518471dcc54f" containerName="revoke-keys" Oct 31 01:32:26.662604 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.662588 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="911a6f16-32c8-44b4-99e1-518471dcc54f" containerName="revoke-keys" Oct 31 01:32:26.665644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.665622 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.704646 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.704616 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.704787 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.704766 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.704835 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.704817 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nk8s\" (UniqueName: \"kubernetes.io/projected/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-kube-api-access-7nk8s\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.717211 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.717192 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:32:26.764540 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.764515 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:32:26.764654 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.764620 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:32:26.766331 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.766312 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r"] Oct 31 01:32:26.805863 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.805842 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nk8s\" (UniqueName: \"kubernetes.io/projected/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-kube-api-access-7nk8s\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.805967 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.805905 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.806014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.805995 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.806613 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.806593 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.808331 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.808315 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.818283 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.818265 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:32:26.865384 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.865359 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nk8s\" (UniqueName: \"kubernetes.io/projected/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-kube-api-access-7nk8s\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:26.978733 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:26.978660 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:27.355381 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:27.355345 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r"] Oct 31 01:32:27.361890 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:32:27.361865 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5134c8_b901_4b23_9b5c_fe44c5f0b4d9.slice/crio-3a6dad9c8022697e8c5f3c3bdbf16faa55e62751e9d7114873b8168720567456 WatchSource:0}: Error finding container 3a6dad9c8022697e8c5f3c3bdbf16faa55e62751e9d7114873b8168720567456: Status 404 returned error can't find the container with id 3a6dad9c8022697e8c5f3c3bdbf16faa55e62751e9d7114873b8168720567456 Oct 31 01:32:27.911010 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:27.910917 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" event={"ID":"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9","Type":"ContainerStarted","Data":"9e029edcddd1cfb498ada8889268c81bf974f59c8ea0453bf2deb0214430805e"} Oct 31 01:32:27.911010 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:27.910984 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" event={"ID":"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9","Type":"ContainerStarted","Data":"3a6dad9c8022697e8c5f3c3bdbf16faa55e62751e9d7114873b8168720567456"} Oct 31 01:32:28.023662 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:28.023603 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" podStartSLOduration=2.023585256 podStartE2EDuration="2.023585256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:28.022449197 +0000 UTC m=+3829.558591503" watchObservedRunningTime="2026-10-31 01:32:28.023585256 +0000 UTC m=+3829.559727562" Oct 31 01:32:28.917612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:28.917573 5719 generic.go:356] "Generic (PLEG): container finished" podID="7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9" containerID="9e029edcddd1cfb498ada8889268c81bf974f59c8ea0453bf2deb0214430805e" exitCode=0 Oct 31 01:32:28.918050 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:28.917658 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" event={"ID":"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9","Type":"ContainerDied","Data":"9e029edcddd1cfb498ada8889268c81bf974f59c8ea0453bf2deb0214430805e"} Oct 31 01:32:30.056110 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.056085 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:30.137180 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.137153 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-openshift-ca\") pod \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " Oct 31 01:32:30.137341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.137267 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-kube-api-access-7nk8s\" (UniqueName: \"kubernetes.io/projected/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-kube-api-access-7nk8s\") pod \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " Oct 31 01:32:30.137341 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.137320 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-xks-ca\") pod \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\" (UID: \"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9\") " Oct 31 01:32:30.137539 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.137514 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-openshift-ca" pod "7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9" (UID: "7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:30.137785 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.137760 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:30.139447 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.139422 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-kube-api-access-7nk8s" pod "7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9" (UID: "7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9"). InnerVolumeSpecName "kube-api-access-7nk8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:30.139566 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.139480 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-xks-ca" pod "7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9" (UID: "7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:30.238839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.238763 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nk8s\" (UniqueName: \"kubernetes.io/projected/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-kube-api-access-7nk8s\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:30.238839 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.238788 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:30.929844 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.929759 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" event={"ID":"7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9","Type":"ContainerDied","Data":"3a6dad9c8022697e8c5f3c3bdbf16faa55e62751e9d7114873b8168720567456"} Oct 31 01:32:30.929844 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.929799 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r" Oct 31 01:32:30.929844 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:30.929815 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6dad9c8022697e8c5f3c3bdbf16faa55e62751e9d7114873b8168720567456" Oct 31 01:32:33.010023 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:32:33.009981 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:32:42.362962 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:42.362930 5719 scope.go:122] "RemoveContainer" containerID="c62de5cf8ba7958be9506c4a35289e40a14f724d38342193c2c91fd556a2891b" Oct 31 01:32:42.372346 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:42.372327 5719 scope.go:122] "RemoveContainer" containerID="7e3fea8caca6c69ef8cc96dc97fac77183f6124a2d9bc6ab60aad59cbab67ab5" Oct 31 01:32:42.380301 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:42.380281 5719 scope.go:122] "RemoveContainer" containerID="c495087998c0e5ea77494d700f6cf127fee8417cc959bf7b9673b4f8a0c144da" Oct 31 01:32:42.858868 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:42.858838 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r"] Oct 31 01:32:42.906260 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:42.906228 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cb9a07-ps62r"] Oct 31 01:32:43.051498 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:43.051456 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:32:44.601475 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:44.601443 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9" path="/var/lib/kubelet/pods/7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9/volumes" Oct 31 01:32:50.968267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:50.968225 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7"] Oct 31 01:32:50.968715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:50.968687 5719 cpu_manager.go:414] "containerMap: removing container" podUID="7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9" containerName="revoke-keys" Oct 31 01:32:50.968715 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:50.968715 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9" containerName="revoke-keys" Oct 31 01:32:50.968841 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:50.968830 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5134c8-b901-4b23-9b5c-fe44c5f0b4d9" containerName="revoke-keys" Oct 31 01:32:50.973682 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:50.973662 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.073201 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.073141 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:32:51.073201 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.073190 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:32:51.073201 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.073198 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:32:51.073557 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.073239 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:32:51.075009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.074990 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7"] Oct 31 01:32:51.144403 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.144364 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831df431-413e-4118-8a0b-fad544dea58c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.144403 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.144409 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831df431-413e-4118-8a0b-fad544dea58c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.144618 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.144504 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plv4k\" (UniqueName: \"kubernetes.io/projected/831df431-413e-4118-8a0b-fad544dea58c-kube-api-access-plv4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.245049 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.244973 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plv4k\" (UniqueName: \"kubernetes.io/projected/831df431-413e-4118-8a0b-fad544dea58c-kube-api-access-plv4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.245202 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.245085 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831df431-413e-4118-8a0b-fad544dea58c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.245202 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.245115 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831df431-413e-4118-8a0b-fad544dea58c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.245747 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.245720 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831df431-413e-4118-8a0b-fad544dea58c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.247502 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.247482 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831df431-413e-4118-8a0b-fad544dea58c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.304449 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.304414 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plv4k\" (UniqueName: \"kubernetes.io/projected/831df431-413e-4118-8a0b-fad544dea58c-kube-api-access-plv4k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.586652 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.586616 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:51.959377 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:51.959297 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7"] Oct 31 01:32:51.966523 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:32:51.966496 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831df431_413e_4118_8a0b_fad544dea58c.slice/crio-4a4bc85bbdf7716a7529cffe87ff93d92f2520e271796a57f220c5d21f8dece6 WatchSource:0}: Error finding container 4a4bc85bbdf7716a7529cffe87ff93d92f2520e271796a57f220c5d21f8dece6: Status 404 returned error can't find the container with id 4a4bc85bbdf7716a7529cffe87ff93d92f2520e271796a57f220c5d21f8dece6 Oct 31 01:32:52.051835 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:52.051797 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" event={"ID":"831df431-413e-4118-8a0b-fad544dea58c","Type":"ContainerStarted","Data":"4a4bc85bbdf7716a7529cffe87ff93d92f2520e271796a57f220c5d21f8dece6"} Oct 31 01:32:53.058668 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:53.058581 5719 generic.go:356] "Generic (PLEG): container finished" podID="831df431-413e-4118-8a0b-fad544dea58c" containerID="5fe886f371ca65ac9b711cbfd209f02b6073f19849c4eca8cf46e595d05f2768" exitCode=0 Oct 31 01:32:53.059081 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:53.058672 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" event={"ID":"831df431-413e-4118-8a0b-fad544dea58c","Type":"ContainerDied","Data":"5fe886f371ca65ac9b711cbfd209f02b6073f19849c4eca8cf46e595d05f2768"} Oct 31 01:32:54.196734 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.196711 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:54.376214 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.376184 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831df431-413e-4118-8a0b-fad544dea58c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831df431-413e-4118-8a0b-fad544dea58c-maas-api-openshift-ca\") pod \"831df431-413e-4118-8a0b-fad544dea58c\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " Oct 31 01:32:54.376412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.376260 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831df431-413e-4118-8a0b-fad544dea58c-kube-api-access-plv4k\" (UniqueName: \"kubernetes.io/projected/831df431-413e-4118-8a0b-fad544dea58c-kube-api-access-plv4k\") pod \"831df431-413e-4118-8a0b-fad544dea58c\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " Oct 31 01:32:54.376412 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.376380 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/831df431-413e-4118-8a0b-fad544dea58c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831df431-413e-4118-8a0b-fad544dea58c-maas-api-xks-ca\") pod \"831df431-413e-4118-8a0b-fad544dea58c\" (UID: \"831df431-413e-4118-8a0b-fad544dea58c\") " Oct 31 01:32:54.376595 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.376571 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831df431-413e-4118-8a0b-fad544dea58c-maas-api-openshift-ca" pod "831df431-413e-4118-8a0b-fad544dea58c" (UID: "831df431-413e-4118-8a0b-fad544dea58c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:54.376716 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.376673 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/831df431-413e-4118-8a0b-fad544dea58c-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:54.378495 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.378471 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831df431-413e-4118-8a0b-fad544dea58c-maas-api-xks-ca" pod "831df431-413e-4118-8a0b-fad544dea58c" (UID: "831df431-413e-4118-8a0b-fad544dea58c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:32:54.378595 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.378556 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831df431-413e-4118-8a0b-fad544dea58c-kube-api-access-plv4k" pod "831df431-413e-4118-8a0b-fad544dea58c" (UID: "831df431-413e-4118-8a0b-fad544dea58c"). InnerVolumeSpecName "kube-api-access-plv4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:54.478161 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.478128 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/831df431-413e-4118-8a0b-fad544dea58c-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:54.478161 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:54.478158 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plv4k\" (UniqueName: \"kubernetes.io/projected/831df431-413e-4118-8a0b-fad544dea58c-kube-api-access-plv4k\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:32:55.071470 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:55.071434 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" event={"ID":"831df431-413e-4118-8a0b-fad544dea58c","Type":"ContainerDied","Data":"4a4bc85bbdf7716a7529cffe87ff93d92f2520e271796a57f220c5d21f8dece6"} Oct 31 01:32:55.071654 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:55.071474 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7" Oct 31 01:32:55.071654 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:32:55.071482 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a4bc85bbdf7716a7529cffe87ff93d92f2520e271796a57f220c5d21f8dece6" Oct 31 01:33:02.533639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:02.533603 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz"] Oct 31 01:33:02.637598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:02.637563 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg"] Oct 31 01:33:03.009026 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:33:03.008989 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:33:07.534327 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:07.534292 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" podUID="3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" containerName="payload-processing" containerID="cri-o://46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729" gracePeriod=25 Oct 31 01:33:07.638846 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:07.638813 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" podUID="59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" containerName="payload-pre-processing" containerID="cri-o://1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72" gracePeriod=25 Oct 31 01:33:13.940046 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:13.940012 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" podUID="3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:33:13.940442 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:13.940012 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" podUID="59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:33:14.867315 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:14.867286 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7"] Oct 31 01:33:14.914936 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:14.914901 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-7f8596-qddp7"] Oct 31 01:33:16.600576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:16.600495 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831df431-413e-4118-8a0b-fad544dea58c" path="/var/lib/kubelet/pods/831df431-413e-4118-8a0b-fad544dea58c/volumes" Oct 31 01:33:18.550166 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:18.550135 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:33:23.763740 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.763683 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s"] Oct 31 01:33:23.764231 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.764216 5719 cpu_manager.go:414] "containerMap: removing container" podUID="831df431-413e-4118-8a0b-fad544dea58c" containerName="revoke-keys" Oct 31 01:33:23.764273 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.764232 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831df431-413e-4118-8a0b-fad544dea58c" containerName="revoke-keys" Oct 31 01:33:23.764319 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.764314 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="831df431-413e-4118-8a0b-fad544dea58c" containerName="revoke-keys" Oct 31 01:33:23.767893 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.767877 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.821417 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.821377 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:33:23.835803 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.835769 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfw4\" (UniqueName: \"kubernetes.io/projected/77fc1332-4cd8-4298-9578-5b80c106bb07-kube-api-access-lmfw4\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.835965 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.835910 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.836009 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.835970 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.868848 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.868814 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:33:23.869025 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.868876 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:33:23.869025 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.868940 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:33:23.870751 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.870726 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s"] Oct 31 01:33:23.936621 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.936590 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.936834 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.936633 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.936834 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.936730 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfw4\" (UniqueName: \"kubernetes.io/projected/77fc1332-4cd8-4298-9578-5b80c106bb07-kube-api-access-lmfw4\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.937342 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.937318 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.939152 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.939131 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:23.940010 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.939982 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" podUID="59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:33:23.940010 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.939995 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" podUID="3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:33:23.995835 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:23.995810 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfw4\" (UniqueName: \"kubernetes.io/projected/77fc1332-4cd8-4298-9578-5b80c106bb07-kube-api-access-lmfw4\") pod \"maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:24.080829 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:24.080799 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:24.460583 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:24.460498 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s"] Oct 31 01:33:24.468613 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:33:24.468585 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fc1332_4cd8_4298_9578_5b80c106bb07.slice/crio-3d07e700179951f090f085beb9593b12254188a75b3d65511de4faa6983f4404 WatchSource:0}: Error finding container 3d07e700179951f090f085beb9593b12254188a75b3d65511de4faa6983f4404: Status 404 returned error can't find the container with id 3d07e700179951f090f085beb9593b12254188a75b3d65511de4faa6983f4404 Oct 31 01:33:25.244835 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:25.244793 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" event={"ID":"77fc1332-4cd8-4298-9578-5b80c106bb07","Type":"ContainerStarted","Data":"5ce5667b6d59ed76ba0cc6092906d4721fd7d28f87c86901e5596a6c82b9fdf8"} Oct 31 01:33:25.245261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:25.244861 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" event={"ID":"77fc1332-4cd8-4298-9578-5b80c106bb07","Type":"ContainerStarted","Data":"3d07e700179951f090f085beb9593b12254188a75b3d65511de4faa6983f4404"} Oct 31 01:33:25.357043 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:25.356996 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" podStartSLOduration=2.356985309 podStartE2EDuration="2.356985309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:25.356222193 +0000 UTC m=+3886.892364499" watchObservedRunningTime="2026-10-31 01:33:25.356985309 +0000 UTC m=+3886.893127649" Oct 31 01:33:26.251275 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:26.251235 5719 generic.go:356] "Generic (PLEG): container finished" podID="77fc1332-4cd8-4298-9578-5b80c106bb07" containerID="5ce5667b6d59ed76ba0cc6092906d4721fd7d28f87c86901e5596a6c82b9fdf8" exitCode=0 Oct 31 01:33:26.251643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:26.251324 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" event={"ID":"77fc1332-4cd8-4298-9578-5b80c106bb07","Type":"ContainerDied","Data":"5ce5667b6d59ed76ba0cc6092906d4721fd7d28f87c86901e5596a6c82b9fdf8"} Oct 31 01:33:27.390430 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.390406 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:27.468800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.468757 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-openshift-ca\") pod \"77fc1332-4cd8-4298-9578-5b80c106bb07\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " Oct 31 01:33:27.469012 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.468821 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77fc1332-4cd8-4298-9578-5b80c106bb07-kube-api-access-lmfw4\" (UniqueName: \"kubernetes.io/projected/77fc1332-4cd8-4298-9578-5b80c106bb07-kube-api-access-lmfw4\") pod \"77fc1332-4cd8-4298-9578-5b80c106bb07\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " Oct 31 01:33:27.469012 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.468890 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-xks-ca\") pod \"77fc1332-4cd8-4298-9578-5b80c106bb07\" (UID: \"77fc1332-4cd8-4298-9578-5b80c106bb07\") " Oct 31 01:33:27.469156 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.469129 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-openshift-ca" pod "77fc1332-4cd8-4298-9578-5b80c106bb07" (UID: "77fc1332-4cd8-4298-9578-5b80c106bb07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:33:27.471115 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.471089 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fc1332-4cd8-4298-9578-5b80c106bb07-kube-api-access-lmfw4" pod "77fc1332-4cd8-4298-9578-5b80c106bb07" (UID: "77fc1332-4cd8-4298-9578-5b80c106bb07"). InnerVolumeSpecName "kube-api-access-lmfw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:27.471268 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.471170 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-xks-ca" pod "77fc1332-4cd8-4298-9578-5b80c106bb07" (UID: "77fc1332-4cd8-4298-9578-5b80c106bb07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:33:27.570164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.570138 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:27.570164 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.570164 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77fc1332-4cd8-4298-9578-5b80c106bb07-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:27.570328 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.570174 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmfw4\" (UniqueName: \"kubernetes.io/projected/77fc1332-4cd8-4298-9578-5b80c106bb07-kube-api-access-lmfw4\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:27.786047 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.785743 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:33:27.873176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.873157 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-plugins-config-volume\") pod \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\" (UID: \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\") " Oct 31 01:33:27.873291 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.873251 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-kube-api-access-4cwqj\" (UniqueName: \"kubernetes.io/projected/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-kube-api-access-4cwqj\") pod \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\" (UID: \"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b\") " Oct 31 01:33:27.873588 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.873551 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-plugins-config-volume" pod "3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" (UID: "3f9e6b31-7248-4f5a-8805-2da63bc3ab4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:33:27.875573 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.875544 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-kube-api-access-4cwqj" pod "3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" (UID: "3f9e6b31-7248-4f5a-8805-2da63bc3ab4b"). InnerVolumeSpecName "kube-api-access-4cwqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:27.877534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.877518 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:33:27.974019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.973982 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-plugins-config-volume\") pod \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\" (UID: \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\") " Oct 31 01:33:27.974019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.974018 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-kube-api-access-wldxc\" (UniqueName: \"kubernetes.io/projected/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-kube-api-access-wldxc\") pod \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\" (UID: \"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8\") " Oct 31 01:33:27.974388 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.974366 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-plugins-config-volume" pod "59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" (UID: "59ec5e5c-0313-4b7d-a8bf-bac04731c7d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:33:27.974445 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.974375 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:27.974445 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.974420 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwqj\" (UniqueName: \"kubernetes.io/projected/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b-kube-api-access-4cwqj\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:27.976294 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:27.976275 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-kube-api-access-wldxc" pod "59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" (UID: "59ec5e5c-0313-4b7d-a8bf-bac04731c7d8"). InnerVolumeSpecName "kube-api-access-wldxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:28.075263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.075221 5719 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-plugins-config-volume\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.075263 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.075254 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wldxc\" (UniqueName: \"kubernetes.io/projected/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8-kube-api-access-wldxc\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:28.228926 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.228886 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq"] Oct 31 01:33:28.242327 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.242297 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j"] Oct 31 01:33:28.263851 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.263814 5719 generic.go:356] "Generic (PLEG): container finished" podID="59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" containerID="1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72" exitCode=0 Oct 31 01:33:28.264014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.263869 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" Oct 31 01:33:28.264014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.263893 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" event={"ID":"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8","Type":"ContainerDied","Data":"1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72"} Oct 31 01:33:28.264014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.263949 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg" event={"ID":"59ec5e5c-0313-4b7d-a8bf-bac04731c7d8","Type":"ContainerDied","Data":"2fa49c8a4356a46cd00ad831698e1caaaef196d93ce394286c5d53be3d8bea4d"} Oct 31 01:33:28.264014 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.263974 5719 scope.go:122] "RemoveContainer" containerID="1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72" Oct 31 01:33:28.265754 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.265729 5719 generic.go:356] "Generic (PLEG): container finished" podID="3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" containerID="46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729" exitCode=0 Oct 31 01:33:28.265873 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.265825 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" Oct 31 01:33:28.265950 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.265866 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" event={"ID":"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b","Type":"ContainerDied","Data":"46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729"} Oct 31 01:33:28.265950 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.265915 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz" event={"ID":"3f9e6b31-7248-4f5a-8805-2da63bc3ab4b","Type":"ContainerDied","Data":"4bab30505daf2017e9a2653525cca3ecb1604892bfd9b151aea46080f7656663"} Oct 31 01:33:28.267505 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.267479 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" event={"ID":"77fc1332-4cd8-4298-9578-5b80c106bb07","Type":"ContainerDied","Data":"3d07e700179951f090f085beb9593b12254188a75b3d65511de4faa6983f4404"} Oct 31 01:33:28.267505 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.267496 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s" Oct 31 01:33:28.267661 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.267527 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d07e700179951f090f085beb9593b12254188a75b3d65511de4faa6983f4404" Oct 31 01:33:28.275267 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.275182 5719 scope.go:122] "RemoveContainer" containerID="1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72" Oct 31 01:33:28.275494 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:33:28.275466 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72\": container with ID starting with 1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72 not found: ID does not exist" containerID="1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72" Oct 31 01:33:28.275612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.275507 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72"} err="failed to get container status \"1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72\": rpc error: code = NotFound desc = could not find container \"1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72\": container with ID starting with 1c8c3c2736c5080379fe8fbdbca39fcec9d0b9d59ca5755bdd3b6389f755ae72 not found: ID does not exist" Oct 31 01:33:28.275612 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.275531 5719 scope.go:122] "RemoveContainer" containerID="46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729" Oct 31 01:33:28.285298 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.285277 5719 scope.go:122] "RemoveContainer" containerID="46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729" Oct 31 01:33:28.285615 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:33:28.285597 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729\": container with ID starting with 46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729 not found: ID does not exist" containerID="46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729" Oct 31 01:33:28.285743 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.285626 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729"} err="failed to get container status \"46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729\": rpc error: code = NotFound desc = could not find container \"46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729\": container with ID starting with 46a5615bf9dda13ca740d61e00139a98db033969c170eb736feabcb4b7206729 not found: ID does not exist" Oct 31 01:33:28.466333 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.466302 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg"] Oct 31 01:33:28.471296 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.471263 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-7f8596-7fcc86599-d9fhg"] Oct 31 01:33:28.581279 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.581242 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz"] Oct 31 01:33:28.584815 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.584788 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-7f8596-75c9bf4785-kmxxz"] Oct 31 01:33:28.601194 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.601166 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" path="/var/lib/kubelet/pods/3f9e6b31-7248-4f5a-8805-2da63bc3ab4b/volumes" Oct 31 01:33:28.601537 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:28.601524 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" path="/var/lib/kubelet/pods/59ec5e5c-0313-4b7d-a8bf-bac04731c7d8/volumes" Oct 31 01:33:32.052879 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:32.052840 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j"] Oct 31 01:33:32.100585 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:32.100561 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx"] Oct 31 01:33:32.101866 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:32.101847 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5be7dc2-pmp5j"] Oct 31 01:33:32.104891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:32.104871 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc24400e-qr4wx"] Oct 31 01:33:32.600560 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:32.600527 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35093d63-9a7e-4570-90c9-cdc67f08bde4" path="/var/lib/kubelet/pods/35093d63-9a7e-4570-90c9-cdc67f08bde4/volumes" Oct 31 01:33:32.600970 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:32.600953 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b66fec2-399e-4fd6-9118-a74cb4f92c68" path="/var/lib/kubelet/pods/4b66fec2-399e-4fd6-9118-a74cb4f92c68/volumes" Oct 31 01:33:33.009332 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:33:33.009254 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:33:39.311940 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.311910 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c"] Oct 31 01:33:39.312413 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312395 5719 cpu_manager.go:414] "containerMap: removing container" podUID="77fc1332-4cd8-4298-9578-5b80c106bb07" containerName="revoke-keys" Oct 31 01:33:39.312479 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312417 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77fc1332-4cd8-4298-9578-5b80c106bb07" containerName="revoke-keys" Oct 31 01:33:39.312479 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312440 5719 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" containerName="payload-processing" Oct 31 01:33:39.312479 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312449 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" containerName="payload-processing" Oct 31 01:33:39.312479 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312476 5719 cpu_manager.go:414] "containerMap: removing container" podUID="59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" containerName="payload-pre-processing" Oct 31 01:33:39.312648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312484 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" containerName="payload-pre-processing" Oct 31 01:33:39.312648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312599 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ec5e5c-0313-4b7d-a8bf-bac04731c7d8" containerName="payload-pre-processing" Oct 31 01:33:39.312648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312626 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="77fc1332-4cd8-4298-9578-5b80c106bb07" containerName="revoke-keys" Oct 31 01:33:39.312648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.312636 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e6b31-7248-4f5a-8805-2da63bc3ab4b" containerName="payload-processing" Oct 31 01:33:39.317562 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.317540 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.372477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.372457 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c"] Oct 31 01:33:39.418234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.418212 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:33:39.418234 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.418212 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:33:39.418376 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.418212 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:33:39.420140 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.420124 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8pzqp\"" Oct 31 01:33:39.478318 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.478295 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kdm\" (UniqueName: \"kubernetes.io/projected/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-kube-api-access-d5kdm\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.478455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.478339 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.478455 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.478446 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.579312 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.579290 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.579426 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.579334 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.579477 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.579459 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kdm\" (UniqueName: \"kubernetes.io/projected/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-kube-api-access-d5kdm\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.580053 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.580036 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.581793 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.581769 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.639581 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.639552 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kdm\" (UniqueName: \"kubernetes.io/projected/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-kube-api-access-d5kdm\") pod \"maas-api-revoke-sub-a6107e17-lrj9c\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:39.929805 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:39.929750 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:40.303348 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:40.303277 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c"] Oct 31 01:33:40.309660 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:33:40.309626 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b8373a_fd28_4ed7_bc68_cdfedb8848ea.slice/crio-2ad0be363cb019ca75964e6b4408d77524b54daa4d2fe12773c8e56949de9a49 WatchSource:0}: Error finding container 2ad0be363cb019ca75964e6b4408d77524b54daa4d2fe12773c8e56949de9a49: Status 404 returned error can't find the container with id 2ad0be363cb019ca75964e6b4408d77524b54daa4d2fe12773c8e56949de9a49 Oct 31 01:33:40.338230 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:40.338193 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" event={"ID":"79b8373a-fd28-4ed7-bc68-cdfedb8848ea","Type":"ContainerStarted","Data":"2ad0be363cb019ca75964e6b4408d77524b54daa4d2fe12773c8e56949de9a49"} Oct 31 01:33:41.345814 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:41.345785 5719 generic.go:356] "Generic (PLEG): container finished" podID="79b8373a-fd28-4ed7-bc68-cdfedb8848ea" containerID="4db42ebc2a09ddd5c50e13fc6992ee1d591de80d9ddfdcda588ebe6a5d946574" exitCode=0 Oct 31 01:33:41.346176 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:41.345826 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" event={"ID":"79b8373a-fd28-4ed7-bc68-cdfedb8848ea","Type":"ContainerDied","Data":"4db42ebc2a09ddd5c50e13fc6992ee1d591de80d9ddfdcda588ebe6a5d946574"} Oct 31 01:33:42.416362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.416336 5719 scope.go:122] "RemoveContainer" containerID="20c05895309444bda393ba57b6f2d0b3942f9410975e0fdfc7c0aeaf7dbb3393" Oct 31 01:33:42.430968 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.430943 5719 scope.go:122] "RemoveContainer" containerID="9811bc9e19f5f8a80d1d1b184f4fbf7f13be13a496184e7cd6cb65d80164f528" Oct 31 01:33:42.440534 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.440508 5719 scope.go:122] "RemoveContainer" containerID="6acbdf8c839b14f17a9b2992ddd1a4e6ef9b2b3628b17f2587caddc6a5e7248f" Oct 31 01:33:42.490159 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.490134 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:42.610748 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.610646 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-kube-api-access-d5kdm\" (UniqueName: \"kubernetes.io/projected/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-kube-api-access-d5kdm\") pod \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " Oct 31 01:33:42.610926 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.610751 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-openshift-ca\") pod \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " Oct 31 01:33:42.610926 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.610825 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-xks-ca\") pod \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\" (UID: \"79b8373a-fd28-4ed7-bc68-cdfedb8848ea\") " Oct 31 01:33:42.611157 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.611130 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-openshift-ca" pod "79b8373a-fd28-4ed7-bc68-cdfedb8848ea" (UID: "79b8373a-fd28-4ed7-bc68-cdfedb8848ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:33:42.612957 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.612932 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-kube-api-access-d5kdm" pod "79b8373a-fd28-4ed7-bc68-cdfedb8848ea" (UID: "79b8373a-fd28-4ed7-bc68-cdfedb8848ea"). InnerVolumeSpecName "kube-api-access-d5kdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:42.613051 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.612992 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-xks-ca" pod "79b8373a-fd28-4ed7-bc68-cdfedb8848ea" (UID: "79b8373a-fd28-4ed7-bc68-cdfedb8848ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:33:42.712109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.712076 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kdm\" (UniqueName: \"kubernetes.io/projected/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-kube-api-access-d5kdm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:42.712109 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.712102 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-openshift-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:42.712283 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:42.712115 5719 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79b8373a-fd28-4ed7-bc68-cdfedb8848ea-maas-api-xks-ca\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.249644 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.249597 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" containerID="cri-o://d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936" gracePeriod=45 Oct 31 01:33:43.262503 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.262477 5719 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" containerID="cri-o://cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6" gracePeriod=45 Oct 31 01:33:43.359043 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.359020 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" Oct 31 01:33:43.359185 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.359039 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6107e17-lrj9c" event={"ID":"79b8373a-fd28-4ed7-bc68-cdfedb8848ea","Type":"ContainerDied","Data":"2ad0be363cb019ca75964e6b4408d77524b54daa4d2fe12773c8e56949de9a49"} Oct 31 01:33:43.359185 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.359083 5719 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad0be363cb019ca75964e6b4408d77524b54daa4d2fe12773c8e56949de9a49" Oct 31 01:33:43.518043 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.518021 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 01:33:43.521116 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.521098 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 01:33:43.619636 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.619607 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-model-cache\") pod \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " Oct 31 01:33:43.619794 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.619659 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tls-certs\" (UniqueName: \"kubernetes.io/secret/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tls-certs\") pod \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " Oct 31 01:33:43.619794 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.619745 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-dshm\") pod \"f591588a-1257-4c5e-a312-63cdcac0d11d\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " Oct 31 01:33:43.619794 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.619774 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-model-cache\") pod \"f591588a-1257-4c5e-a312-63cdcac0d11d\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " Oct 31 01:33:43.619994 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.619828 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-home\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-home\") pod \"f591588a-1257-4c5e-a312-63cdcac0d11d\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " Oct 31 01:33:43.619994 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.619884 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f591588a-1257-4c5e-a312-63cdcac0d11d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f591588a-1257-4c5e-a312-63cdcac0d11d-tls-certs\") pod \"f591588a-1257-4c5e-a312-63cdcac0d11d\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " Oct 31 01:33:43.619994 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.619907 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-model-cache" pod "18be2d8e-96aa-4dbd-81b2-2f42df8f7083" (UID: "18be2d8e-96aa-4dbd-81b2-2f42df8f7083"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.619994 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.619913 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-dshm\") pod \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " Oct 31 01:33:43.620233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620009 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f591588a-1257-4c5e-a312-63cdcac0d11d-kube-api-access-nvvxf\" (UniqueName: \"kubernetes.io/projected/f591588a-1257-4c5e-a312-63cdcac0d11d-kube-api-access-nvvxf\") pod \"f591588a-1257-4c5e-a312-63cdcac0d11d\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " Oct 31 01:33:43.620233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620069 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-kube-api-access-fz2sd\" (UniqueName: \"kubernetes.io/projected/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-kube-api-access-fz2sd\") pod \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " Oct 31 01:33:43.620233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620107 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-home" pod "f591588a-1257-4c5e-a312-63cdcac0d11d" (UID: "f591588a-1257-4c5e-a312-63cdcac0d11d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.620233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620125 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-model-cache" pod "f591588a-1257-4c5e-a312-63cdcac0d11d" (UID: "f591588a-1257-4c5e-a312-63cdcac0d11d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.620233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620148 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-home\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-home\") pod \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " Oct 31 01:33:43.620233 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620185 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-tmp-dir\") pod \"f591588a-1257-4c5e-a312-63cdcac0d11d\" (UID: \"f591588a-1257-4c5e-a312-63cdcac0d11d\") " Oct 31 01:33:43.620558 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620251 5719 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tmp-dir\") pod \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\" (UID: \"18be2d8e-96aa-4dbd-81b2-2f42df8f7083\") " Oct 31 01:33:43.621019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620666 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-tmp-dir" pod "f591588a-1257-4c5e-a312-63cdcac0d11d" (UID: "f591588a-1257-4c5e-a312-63cdcac0d11d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.621019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620830 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.621019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620852 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.621019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620868 5719 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-model-cache\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.621019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.620978 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-home" pod "18be2d8e-96aa-4dbd-81b2-2f42df8f7083" (UID: "18be2d8e-96aa-4dbd-81b2-2f42df8f7083"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.621019 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.621016 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tmp-dir" pod "18be2d8e-96aa-4dbd-81b2-2f42df8f7083" (UID: "18be2d8e-96aa-4dbd-81b2-2f42df8f7083"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.622603 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.622563 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f591588a-1257-4c5e-a312-63cdcac0d11d-tls-certs" pod "f591588a-1257-4c5e-a312-63cdcac0d11d" (UID: "f591588a-1257-4c5e-a312-63cdcac0d11d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:33:43.622732 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.622595 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-kube-api-access-fz2sd" pod "18be2d8e-96aa-4dbd-81b2-2f42df8f7083" (UID: "18be2d8e-96aa-4dbd-81b2-2f42df8f7083"). InnerVolumeSpecName "kube-api-access-fz2sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:43.622732 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.622667 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-dshm" pod "18be2d8e-96aa-4dbd-81b2-2f42df8f7083" (UID: "18be2d8e-96aa-4dbd-81b2-2f42df8f7083"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.622732 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.622673 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tls-certs" pod "18be2d8e-96aa-4dbd-81b2-2f42df8f7083" (UID: "18be2d8e-96aa-4dbd-81b2-2f42df8f7083"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:33:43.622943 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.622923 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f591588a-1257-4c5e-a312-63cdcac0d11d-kube-api-access-nvvxf" pod "f591588a-1257-4c5e-a312-63cdcac0d11d" (UID: "f591588a-1257-4c5e-a312-63cdcac0d11d"). InnerVolumeSpecName "kube-api-access-nvvxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:33:43.622943 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.622932 5719 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-dshm" pod "f591588a-1257-4c5e-a312-63cdcac0d11d" (UID: "f591588a-1257-4c5e-a312-63cdcac0d11d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:33:43.721587 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721555 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvxf\" (UniqueName: \"kubernetes.io/projected/f591588a-1257-4c5e-a312-63cdcac0d11d-kube-api-access-nvvxf\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.721587 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721582 5719 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz2sd\" (UniqueName: \"kubernetes.io/projected/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-kube-api-access-fz2sd\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.721587 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721593 5719 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-home\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.721838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721602 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.721838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721611 5719 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tmp-dir\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.721838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721620 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.721838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721628 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f591588a-1257-4c5e-a312-63cdcac0d11d-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.721838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721637 5719 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f591588a-1257-4c5e-a312-63cdcac0d11d-tls-certs\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:43.721838 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:43.721644 5719 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18be2d8e-96aa-4dbd-81b2-2f42df8f7083-dshm\") on node \"ip-10-0-129-8.us-west-2.compute.internal\" DevicePath \"\"" Oct 31 01:33:44.365883 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.365850 5719 generic.go:356] "Generic (PLEG): container finished" podID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerID="d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936" exitCode=0 Oct 31 01:33:44.366064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.365898 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" Oct 31 01:33:44.366064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.365931 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" event={"ID":"f591588a-1257-4c5e-a312-63cdcac0d11d","Type":"ContainerDied","Data":"d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936"} Oct 31 01:33:44.366064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.365976 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" event={"ID":"f591588a-1257-4c5e-a312-63cdcac0d11d","Type":"ContainerDied","Data":"e76b4e568e639d299bb920df82269a94e75ebf12b77d98d7e706be19c3121b6b"} Oct 31 01:33:44.366064 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.365989 5719 scope.go:122] "RemoveContainer" containerID="d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936" Oct 31 01:33:44.367515 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.367491 5719 generic.go:356] "Generic (PLEG): container finished" podID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerID="cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6" exitCode=0 Oct 31 01:33:44.367639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.367531 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" event={"ID":"18be2d8e-96aa-4dbd-81b2-2f42df8f7083","Type":"ContainerDied","Data":"cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6"} Oct 31 01:33:44.367639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.367543 5719 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" Oct 31 01:33:44.367639 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.367554 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j" event={"ID":"18be2d8e-96aa-4dbd-81b2-2f42df8f7083","Type":"ContainerDied","Data":"6592a90e2b48e5599c48e36bf932c173668dc573c12a66bbdab9c31f4377890c"} Oct 31 01:33:44.376546 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.376364 5719 scope.go:122] "RemoveContainer" containerID="d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936" Oct 31 01:33:44.376667 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:33:44.376647 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936\": container with ID starting with d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936 not found: ID does not exist" containerID="d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936" Oct 31 01:33:44.376774 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.376681 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936"} err="failed to get container status \"d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936\": rpc error: code = NotFound desc = could not find container \"d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936\": container with ID starting with d7ac915ca8fed27eae6d5087175ddea80597a7b6a9c45f17e74bf31521bc5936 not found: ID does not exist" Oct 31 01:33:44.376774 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.376740 5719 scope.go:122] "RemoveContainer" containerID="cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6" Oct 31 01:33:44.384862 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.384843 5719 scope.go:122] "RemoveContainer" containerID="cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6" Oct 31 01:33:44.385098 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:33:44.385080 5719 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6\": container with ID starting with cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6 not found: ID does not exist" containerID="cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6" Oct 31 01:33:44.385200 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.385103 5719 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6"} err="failed to get container status \"cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6\": rpc error: code = NotFound desc = could not find container \"cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6\": container with ID starting with cf9e00e7dc4c4a2aca561291b74b255df7ffbac4438918ea359a60a2b50b77b6 not found: ID does not exist" Oct 31 01:33:44.489821 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.489795 5719 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:33:44.530397 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.530367 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j"] Oct 31 01:33:44.577760 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.577736 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/premium-w0-8bec73-kserve-6cd86f4f94-tfh9j"] Oct 31 01:33:44.601591 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.601563 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" path="/var/lib/kubelet/pods/18be2d8e-96aa-4dbd-81b2-2f42df8f7083/volumes" Oct 31 01:33:44.682552 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.682497 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq"] Oct 31 01:33:44.689063 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:44.689043 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8bec73/distinct-2-w0-8bec73-kserve-58cc48c598-rcsxq"] Oct 31 01:33:46.599938 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:46.599906 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" path="/var/lib/kubelet/pods/f591588a-1257-4c5e-a312-63cdcac0d11d/volumes" Oct 31 01:33:55.985438 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:55.985397 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:33:56.776112 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:56.776074 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s"] Oct 31 01:33:56.823470 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:56.823437 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-8bec73-ctx7s"] Oct 31 01:33:58.601362 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:33:58.601332 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77fc1332-4cd8-4298-9578-5b80c106bb07" path="/var/lib/kubelet/pods/77fc1332-4cd8-4298-9578-5b80c106bb07/volumes" Oct 31 01:34:03.009935 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:34:03.009900 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:34:06.897549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:06.897520 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xq2lx_371050b9-e409-4c0d-83f1-b198aef0d8fc/kuadrant-console-plugin/0.log" Oct 31 01:34:08.513673 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:08.513639 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zh575_d3ed9194-eec5-4d49-b8ae-e086c7079bb8/limitador/0.log" Oct 31 01:34:09.656723 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:09.656679 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-zh575"] Oct 31 01:34:09.999151 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:09.999082 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vjnhm_3ff32f5a-d8bd-42e6-a017-67721b1ccb53/manager/0.log" Oct 31 01:34:10.506397 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:10.506363 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rmwbn_6afc4a25-54af-4485-a899-ca98fc0f0944/manager/0.log" Oct 31 01:34:11.559835 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:11.559784 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zcp2r_85ba94f8-58b4-44db-8f31-984b32325ccb/manager/0.log" Oct 31 01:34:11.845922 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:11.845882 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5cdlq_23ac674f-79e0-434e-9c62-b8e6beb611f9/proxy/0.log" Oct 31 01:34:24.150555 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:24.150523 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vjnhm_3ff32f5a-d8bd-42e6-a017-67721b1ccb53/manager/0.log" Oct 31 01:34:24.663138 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:24.663105 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rmwbn_6afc4a25-54af-4485-a899-ca98fc0f0944/manager/0.log" Oct 31 01:34:25.624326 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:25.624294 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zcp2r_85ba94f8-58b4-44db-8f31-984b32325ccb/manager/0.log" Oct 31 01:34:25.885202 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:25.885129 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5cdlq_23ac674f-79e0-434e-9c62-b8e6beb611f9/proxy/0.log" Oct 31 01:34:27.408849 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:27.408823 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-51e96aed-z9x9s_28af5f7b-4f8d-4a73-b93b-e223ce8ebe7b/revoke-keys/0.log" Oct 31 01:34:27.859660 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:27.859632 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a6107e17-lrj9c_79b8373a-fd28-4ed7-bc68-cdfedb8848ea/revoke-keys/0.log" Oct 31 01:34:28.077710 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:28.077670 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b678854d-w47qx_269c38ce-d974-4242-aa15-21af8c116ddc/revoke-keys/0.log" Oct 31 01:34:28.538119 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:28.538090 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f61e8de3-jdqff_6e05e9bf-502a-4690-82d6-9935089983a2/revoke-keys/0.log" Oct 31 01:34:31.478614 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:31.478586 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xq2lx_371050b9-e409-4c0d-83f1-b198aef0d8fc/kuadrant-console-plugin/0.log" Oct 31 01:34:32.797418 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:32.797385 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zh575_d3ed9194-eec5-4d49-b8ae-e086c7079bb8/limitador/0.log" Oct 31 01:34:33.009123 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:34:33.009090 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:34:34.715739 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:34.715705 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67865c945c-w5smw_40461298-1154-443e-bd2f-b0dbd0ed9537/kube-auth-proxy/0.log" Oct 31 01:34:34.949523 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:34.949484 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-fk527_db26ddf5-d18b-4d40-8107-a76902839346/istio-proxy/0.log" Oct 31 01:34:42.480232 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:42.480202 5719 scope.go:122] "RemoveContainer" containerID="aadfb99169c3c6c616483fd2b89b89ecd74a5c688a58a6fb0b0854380e90025b" Oct 31 01:34:42.489409 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:42.489390 5719 scope.go:122] "RemoveContainer" containerID="4416a09eff9863dcfea958f023edac33645b46c2ad6bcacadca79cc348b6c15d" Oct 31 01:34:42.497526 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:42.497509 5719 scope.go:122] "RemoveContainer" containerID="7e9e0706db08fc666eb7d0ba7f7eb23a8e7ff93cdb192f1880c9e38f511512de" Oct 31 01:34:42.505269 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:34:42.505255 5719 scope.go:122] "RemoveContainer" containerID="9f1851ad9306a12a4ea5926f55aeede88783047e051a0e146d2e8d7b33e48fed" Oct 31 01:35:01.108502 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:01.108475 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ht56p_8c091a1b-9f15-4dcf-a55a-4d2c07ddcdc5/global-pull-secret-syncer/0.log" Oct 31 01:35:01.666981 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:01.666955 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vjnhm_3ff32f5a-d8bd-42e6-a017-67721b1ccb53/manager/0.log" Oct 31 01:35:02.434226 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:02.434186 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rmwbn_6afc4a25-54af-4485-a899-ca98fc0f0944/manager/0.log" Oct 31 01:35:03.008748 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:35:03.008708 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:35:03.583228 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:03.583198 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zcp2r_85ba94f8-58b4-44db-8f31-984b32325ccb/manager/0.log" Oct 31 01:35:03.694588 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:03.694563 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5cdlq_23ac674f-79e0-434e-9c62-b8e6beb611f9/proxy/0.log" Oct 31 01:35:03.856643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:03.856565 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vjnhm_3ff32f5a-d8bd-42e6-a017-67721b1ccb53/manager/0.log" Oct 31 01:35:04.098089 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:04.098059 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m98mn_b75cde1b-aaee-46c8-b956-dcecf3b6afbf/konnectivity-agent/0.log" Oct 31 01:35:04.629154 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:04.623315 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rmwbn_6afc4a25-54af-4485-a899-ca98fc0f0944/manager/0.log" Oct 31 01:35:04.987652 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:04.987589 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-8.us-west-2.compute.internal_eaf6c79eda58382b69cd6ec1e1ffc85d/haproxy/0.log" Oct 31 01:35:06.592666 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:06.592634 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zcp2r_85ba94f8-58b4-44db-8f31-984b32325ccb/manager/0.log" Oct 31 01:35:06.945900 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:06.945830 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vjnhm_3ff32f5a-d8bd-42e6-a017-67721b1ccb53/manager/0.log" Oct 31 01:35:07.052483 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:07.052461 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5cdlq_23ac674f-79e0-434e-9c62-b8e6beb611f9/proxy/0.log" Oct 31 01:35:07.702656 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:07.702627 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rmwbn_6afc4a25-54af-4485-a899-ca98fc0f0944/manager/0.log" Oct 31 01:35:09.013501 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:09.013474 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zcp2r_85ba94f8-58b4-44db-8f31-984b32325ccb/manager/0.log" Oct 31 01:35:09.197915 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:09.197891 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5cdlq_23ac674f-79e0-434e-9c62-b8e6beb611f9/proxy/0.log" Oct 31 01:35:10.053873 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:10.053842 5719 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx"] Oct 31 01:35:10.101302 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:10.101275 5719 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b678854d-w47qx"] Oct 31 01:35:10.528676 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:10.528645 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vjnhm_3ff32f5a-d8bd-42e6-a017-67721b1ccb53/manager/0.log" Oct 31 01:35:10.600703 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:10.600668 5719 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269c38ce-d974-4242-aa15-21af8c116ddc" path="/var/lib/kubelet/pods/269c38ce-d974-4242-aa15-21af8c116ddc/volumes" Oct 31 01:35:11.209614 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.209582 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rmwbn_6afc4a25-54af-4485-a899-ca98fc0f0944/manager/0.log" Oct 31 01:35:11.623055 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623027 5719 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2"] Oct 31 01:35:11.623402 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623391 5719 cpu_manager.go:414] "containerMap: removing container" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" Oct 31 01:35:11.623460 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623404 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" Oct 31 01:35:11.623460 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623413 5719 cpu_manager.go:414] "containerMap: removing container" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" Oct 31 01:35:11.623460 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623418 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" Oct 31 01:35:11.623460 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623442 5719 cpu_manager.go:414] "containerMap: removing container" podUID="79b8373a-fd28-4ed7-bc68-cdfedb8848ea" containerName="revoke-keys" Oct 31 01:35:11.623460 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623447 5719 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b8373a-fd28-4ed7-bc68-cdfedb8848ea" containerName="revoke-keys" Oct 31 01:35:11.623627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623508 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="18be2d8e-96aa-4dbd-81b2-2f42df8f7083" containerName="main" Oct 31 01:35:11.623627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623517 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="f591588a-1257-4c5e-a312-63cdcac0d11d" containerName="main" Oct 31 01:35:11.623627 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.623527 5719 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b8373a-fd28-4ed7-bc68-cdfedb8848ea" containerName="revoke-keys" Oct 31 01:35:11.626598 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.626579 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.677058 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.677032 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dczd8\"/\"kube-root-ca.crt\"" Oct 31 01:35:11.677167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.677064 5719 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dczd8\"/\"openshift-service-ca.crt\"" Oct 31 01:35:11.677167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.677083 5719 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dczd8\"/\"default-dockercfg-b7mvt\"" Oct 31 01:35:11.726604 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.726578 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2"] Oct 31 01:35:11.730576 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.730551 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-podres\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.730705 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.730673 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-lib-modules\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.730781 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.730752 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jrn\" (UniqueName: \"kubernetes.io/projected/a1e5b0c6-a80f-488c-bb4c-776f608e2778-kube-api-access-x7jrn\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.730891 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.730870 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-sys\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.730933 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.730923 5719 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-proc\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832042 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832022 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jrn\" (UniqueName: \"kubernetes.io/projected/a1e5b0c6-a80f-488c-bb4c-776f608e2778-kube-api-access-x7jrn\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832099 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-sys\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832146 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832127 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-proc\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832150 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-podres\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832228 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-sys\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832232 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-podres\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832261 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832248 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-proc\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832421 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832280 5719 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-lib-modules\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.832421 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.832385 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a1e5b0c6-a80f-488c-bb4c-776f608e2778-lib-modules\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.890554 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.890502 5719 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jrn\" (UniqueName: \"kubernetes.io/projected/a1e5b0c6-a80f-488c-bb4c-776f608e2778-kube-api-access-x7jrn\") pod \"perf-node-gather-daemonset-rd7h2\" (UID: \"a1e5b0c6-a80f-488c-bb4c-776f608e2778\") " pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:11.938865 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:11.938843 5719 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:12.295183 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.295102 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vjnhm_3ff32f5a-d8bd-42e6-a017-67721b1ccb53/manager/0.log" Oct 31 01:35:12.316800 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.316766 5719 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2"] Oct 31 01:35:12.323908 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.323883 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zcp2r_85ba94f8-58b4-44db-8f31-984b32325ccb/manager/0.log" Oct 31 01:35:12.324206 ip-10-0-129-8 kubenswrapper[5719]: W1031 01:35:12.324185 5719 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda1e5b0c6_a80f_488c_bb4c_776f608e2778.slice/crio-7a3a27013820b63d4411f0306962bb74be34f479cb15a3db77f432bf76671031 WatchSource:0}: Error finding container 7a3a27013820b63d4411f0306962bb74be34f479cb15a3db77f432bf76671031: Status 404 returned error can't find the container with id 7a3a27013820b63d4411f0306962bb74be34f479cb15a3db77f432bf76671031 Oct 31 01:35:12.456705 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.456669 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5cdlq_23ac674f-79e0-434e-9c62-b8e6beb611f9/proxy/0.log" Oct 31 01:35:12.717650 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.717628 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xq2lx_371050b9-e409-4c0d-83f1-b198aef0d8fc/kuadrant-console-plugin/0.log" Oct 31 01:35:12.839771 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.839736 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" event={"ID":"a1e5b0c6-a80f-488c-bb4c-776f608e2778","Type":"ContainerStarted","Data":"f407c702bbfe4cb9fcd4b0ce13627abdd3e1f6c623e03e2cfe7a08c8380fa7b5"} Oct 31 01:35:12.839907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.839796 5719 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" event={"ID":"a1e5b0c6-a80f-488c-bb4c-776f608e2778","Type":"ContainerStarted","Data":"7a3a27013820b63d4411f0306962bb74be34f479cb15a3db77f432bf76671031"} Oct 31 01:35:12.839907 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.839850 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:12.970606 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:12.970532 5719 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" podStartSLOduration=1.9705180709999999 podStartE2EDuration="1.970518071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:12.970288972 +0000 UTC m=+3994.506431277" watchObservedRunningTime="2026-10-31 01:35:12.970518071 +0000 UTC m=+3994.506660375" Oct 31 01:35:13.006602 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:13.006574 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rmwbn_6afc4a25-54af-4485-a899-ca98fc0f0944/manager/0.log" Oct 31 01:35:14.143940 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:14.143914 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zcp2r_85ba94f8-58b4-44db-8f31-984b32325ccb/manager/0.log" Oct 31 01:35:14.227920 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:14.227885 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zh575_d3ed9194-eec5-4d49-b8ae-e086c7079bb8/limitador/0.log" Oct 31 01:35:14.280403 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:14.280382 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5cdlq_23ac674f-79e0-434e-9c62-b8e6beb611f9/proxy/0.log" Oct 31 01:35:15.654617 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:15.654584 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vjnhm_3ff32f5a-d8bd-42e6-a017-67721b1ccb53/manager/0.log" Oct 31 01:35:16.321648 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:16.321619 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rmwbn_6afc4a25-54af-4485-a899-ca98fc0f0944/manager/0.log" Oct 31 01:35:17.346456 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:17.346415 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_218ebb1b-681e-49dc-b635-83704e5a17eb/alertmanager/0.log" Oct 31 01:35:17.444912 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:17.444884 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-zcp2r_85ba94f8-58b4-44db-8f31-984b32325ccb/manager/0.log" Oct 31 01:35:17.578769 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:17.578735 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-5cdlq_23ac674f-79e0-434e-9c62-b8e6beb611f9/proxy/0.log" Oct 31 01:35:17.770167 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:17.770071 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_218ebb1b-681e-49dc-b635-83704e5a17eb/config-reloader/0.log" Oct 31 01:35:18.186337 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:18.186305 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_218ebb1b-681e-49dc-b635-83704e5a17eb/kube-rbac-proxy-web/0.log" Oct 31 01:35:18.601353 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:18.601328 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_218ebb1b-681e-49dc-b635-83704e5a17eb/kube-rbac-proxy/0.log" Oct 31 01:35:18.783562 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:18.783538 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67865c945c-w5smw_40461298-1154-443e-bd2f-b0dbd0ed9537/kube-auth-proxy/0.log" Oct 31 01:35:18.855749 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:18.855677 5719 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dczd8/perf-node-gather-daemonset-rd7h2" Oct 31 01:35:18.999903 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:18.999880 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-fk527_db26ddf5-d18b-4d40-8107-a76902839346/istio-proxy/0.log" Oct 31 01:35:19.015179 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:19.015160 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_218ebb1b-681e-49dc-b635-83704e5a17eb/kube-rbac-proxy-metric/0.log" Oct 31 01:35:19.428556 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:19.428535 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_218ebb1b-681e-49dc-b635-83704e5a17eb/prom-label-proxy/0.log" Oct 31 01:35:19.842066 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:19.842045 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_218ebb1b-681e-49dc-b635-83704e5a17eb/init-config-reloader/0.log" Oct 31 01:35:20.700264 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:20.700237 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9klj5_75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/kube-state-metrics/0.log" Oct 31 01:35:21.108813 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:21.108777 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9klj5_75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/kube-rbac-proxy-main/0.log" Oct 31 01:35:21.527850 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:21.527775 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9klj5_75963a1c-a5af-4c12-8adc-f6c5b9aa2e39/kube-rbac-proxy-self/0.log" Oct 31 01:35:21.946147 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:21.946120 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67865c945c-w5smw_40461298-1154-443e-bd2f-b0dbd0ed9537/kube-auth-proxy/0.log" Oct 31 01:35:21.950643 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:21.950626 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d46547967-pkq4f_5f65619e-5f82-4313-8a35-572fbece3755/metrics-server/0.log" Oct 31 01:35:22.156197 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:22.156171 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-fk527_db26ddf5-d18b-4d40-8107-a76902839346/istio-proxy/0.log" Oct 31 01:35:22.365821 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:22.365794 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hxlqh_fda3848b-9d0b-4832-8789-862920d484b8/monitoring-plugin/0.log" Oct 31 01:35:24.070354 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:24.070316 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mqb6s_d142f31b-3461-4f5a-93e1-088ce6511100/node-exporter/0.log" Oct 31 01:35:24.489387 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:24.489310 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mqb6s_d142f31b-3461-4f5a-93e1-088ce6511100/kube-rbac-proxy/0.log" Oct 31 01:35:24.910714 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:24.910673 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mqb6s_d142f31b-3461-4f5a-93e1-088ce6511100/init-textfile/0.log" Oct 31 01:35:25.911076 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:25.911046 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xq2lx_371050b9-e409-4c0d-83f1-b198aef0d8fc/kuadrant-console-plugin/0.log" Oct 31 01:35:26.606068 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:26.606042 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lspcv_10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/kube-rbac-proxy-main/0.log" Oct 31 01:35:26.963304 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:26.963242 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zh575_d3ed9194-eec5-4d49-b8ae-e086c7079bb8/limitador/0.log" Oct 31 01:35:27.023887 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:27.023864 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lspcv_10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/kube-rbac-proxy-self/0.log" Oct 31 01:35:27.437336 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:27.437308 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lspcv_10c4027c-b9b6-4a0d-a6d0-66f5a76004bf/openshift-state-metrics/0.log" Oct 31 01:35:27.867574 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:27.867543 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7166d6c5-eb2d-4661-9315-e6ecfee88bd2/prometheus/0.log" Oct 31 01:35:28.281049 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:28.280981 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7166d6c5-eb2d-4661-9315-e6ecfee88bd2/config-reloader/0.log" Oct 31 01:35:28.694321 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:28.694293 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7166d6c5-eb2d-4661-9315-e6ecfee88bd2/thanos-sidecar/0.log" Oct 31 01:35:29.107783 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:29.107755 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7166d6c5-eb2d-4661-9315-e6ecfee88bd2/kube-rbac-proxy-web/0.log" Oct 31 01:35:29.525841 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:29.525771 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7166d6c5-eb2d-4661-9315-e6ecfee88bd2/kube-rbac-proxy/0.log" Oct 31 01:35:29.563582 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:29.563555 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xq2lx_371050b9-e409-4c0d-83f1-b198aef0d8fc/kuadrant-console-plugin/0.log" Oct 31 01:35:29.941022 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:29.940996 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7166d6c5-eb2d-4661-9315-e6ecfee88bd2/kube-rbac-proxy-thanos/0.log" Oct 31 01:35:30.367777 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:30.367745 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7166d6c5-eb2d-4661-9315-e6ecfee88bd2/init-config-reloader/0.log" Oct 31 01:35:30.592132 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:30.592109 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-zh575_d3ed9194-eec5-4d49-b8ae-e086c7079bb8/limitador/0.log" Oct 31 01:35:32.038309 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:32.038286 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5597468789-lbx94_7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b/telemeter-client/0.log" Oct 31 01:35:32.451270 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:32.451250 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5597468789-lbx94_7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b/reload/0.log" Oct 31 01:35:32.864652 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:32.864623 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5597468789-lbx94_7c37daf4-abfa-4b1d-a4ab-fd5d7eadd39b/kube-rbac-proxy/0.log" Oct 31 01:35:33.009377 ip-10-0-129-8 kubenswrapper[5719]: E1031 01:35:33.009345 5719 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-47ws4" containerName="extractor" Oct 31 01:35:33.284724 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:33.284631 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dcb67b5b7-gwgv4_d0910c96-18d0-49b2-861d-0ec2a5793047/thanos-query/0.log" Oct 31 01:35:33.697708 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:33.697651 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dcb67b5b7-gwgv4_d0910c96-18d0-49b2-861d-0ec2a5793047/kube-rbac-proxy-web/0.log" Oct 31 01:35:34.110719 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:34.110674 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dcb67b5b7-gwgv4_d0910c96-18d0-49b2-861d-0ec2a5793047/kube-rbac-proxy/0.log" Oct 31 01:35:34.533415 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:34.533349 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dcb67b5b7-gwgv4_d0910c96-18d0-49b2-861d-0ec2a5793047/prom-label-proxy/0.log" Oct 31 01:35:34.956536 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:34.956507 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dcb67b5b7-gwgv4_d0910c96-18d0-49b2-861d-0ec2a5793047/kube-rbac-proxy-rules/0.log" Oct 31 01:35:35.393196 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:35.393173 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6dcb67b5b7-gwgv4_d0910c96-18d0-49b2-861d-0ec2a5793047/kube-rbac-proxy-metrics/0.log" Oct 31 01:35:36.137549 ip-10-0-129-8 kubenswrapper[5719]: I1031 01:35:36.137519 5719 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67865c945c-w5smw_40461298-1154-443e-bd2f-b0dbd0ed9537/kube-auth-proxy/0.log"